Improve Data Engineering Hand Book?
Thoughts about how to improve Data Engineering Hand Book?
- This is project and I do need to get feedback, summary of the book
- This is markdown project, it is possible to be scanned and loadded into LLMs
- Using Ollama to integrate and work with this project
-
I built this thought note and got shared the image above from network, it motivated me a lot of building this knowledge base.
-
From the diagram, we can basic understand how the [traning knowledge base work].
Changes
- [2024-04-05] Init version and tried bot
- [2024-04-07] Init note
- [2024-04-13] Update diagram of work
Key Notes
:fontawesome-brands-youtube:{ .youtube } Youtube Video for Demo → Enhancing Data Engineering Book using LLM App
Play the game
Step 1: Setup Application
Start every applition,
Download Vector database for storing and indexing docs
Step 2: Init Vector Database
Load the book location into the Application (local only).
This application is use LanceDb for demonstration, and starting to load into embedding model before indexing the docs.
Step 3: Ask Bot
I wanted to get the bot opinions of my works to see if it meets the goal?
And, How to tell the audiences how to get starting with this book.
Step 4: Bot Response
The bot has generated the summary, and how to start with the book.
Futhermore, bot suggesteda few of actions that help me to improve the book.
Referrence
Conclusion
Becareful if you wanted to you and train anything on laptop
- Good try and have fun!
- Not well suitabe for
chiplaptop model (I am using Air M1 Base) :smile: - Can be deploy on Server and create a good recommendation by calling API hook.
- Imaging how Notion AI works