Maintelyd

Public Data Discoveries, from Messy-Unstructured Sources into Easy-Tabular Data with Historical Timestamps. Ready for your further analysis, research, even feed to your agents.

Comment

Inspiration

I am data analyst with 8 years experience. I work with structured tabular data every day, with BigQuery platform. With tabular data, I could create many data products, either dashboard/adhoc analysis. And that is the usual common data structure setting in the company.

Outside company, especially in Internet, the data amount is vast, yet unstructured. News, articles, social media posting, journals, etc. are also data, that we send and receive every day publicly. What if, we could utilize & optimize those scattered data sources for beneficial purposes.

It might be difficult to think "kind of beneficial purposes" from those scattered messy data. So I will take first case, which is global product price. We believe, that our groceries (for example, beef, milk, egg, vegetables, fruit) are cheaper . But are we confident enough ? What if we find , that some of our groceries price in our country are actually more expensive than on other countries ? If that case happens, then we will do anything, either by submitting feedbacks to government/ research on ourself capability on how to produce food with lesser cost/ research on best practice/ etc., to drive the groceries price down. However, global product price sources are truly scattered and vast. Also, we cannot manually click on many E-commerce sites and merchant online stores and copy paste the product name and prices manually into spreadsheet.

Those above bring me inspiration : how could I utilize current emerging technologies, which are agentic workflows and automation, to discover unstructured scattered global product prices and turn it into simple tabular data.

What it does

Maintelyd discovers global product price through 3 ways :

  1. web search on specific topics
  2. deep data extraction on targeted URLs
  3. Google Shopping API

Then the discovered unstructured data was parsed into product name, price, unit quantity, unit measurement scale, rating and review count (from Google Shopping only), source, and country. The data was enhanced with price currency (USD, EUR, CHF, JPY, CNY, SGD, AUD). Then the data is visualized through dashboards consist of charts, table, and raw data. Through charts and tables, user will be able to compare product prices across all countries instantly in many many product categories. Through raw data, user will be able to have access on more detailed data dimension, and conduct analysis on their own.

Here are 20 countries currently covered : United States, Brazil, Argentina, Chile, United Kingdom, France, Germany, Algeria, Tanzania, South Africa, Saudi Arabia, Iraq, Russia, Japan, China, India, Thailand, Indonesia, Singapore, Australia, New Zealand.

Recently Maintelyd provide "Product Price Master Agent", where user will be able to do any product price analysis topic to our database, through agent's understanding to human language and adaptive SQL coding.

How we built it

1. Agentic Workflow

Agentic Workflow is important to orchestrate the line process to update database so new product price data keep coming & recorded. We built with LangGraph. The line process from beginning to the end are :

A. product price discovery

B. data parsing

C. product name translation from local language to English

D. product category mapping

E. measurement scale and quantity standardization

F. embed price currency conversion

G. final tabular data insertion to Supabase

1 A. Product Price Discovery

Maintelyd discovers global product price through 3 ways :

  1. web search on specific topics : we use OpenAI GPT-5.4, tool = web_search.
  2. deep data extraction on targeted URLs : we use Tavily Extract.
  3. Google Shopping API : we use Serper API.

1 B. Data Parsing

The unstructured text data is parsed to extract these information : product name, price , quantity, measurement scale, source, source date, rating and review count (from Google Shopping only). We use mixed regex cleaning and OpenAI GPT-5-mini to maintain lower cost while meeting good parsing quality.

1 C. Product Name Translation

Because we discover product prices globally, product names from countries with non-English speakers (Japan, China, Thailand, Russia, Saudi Arabia, South Africa, Brazil, Argentina, etc.) are in their native language. In order all products are understandable multi-countries, we add new information, which is product name in English. We use OpenAI GPT-5-mini to translate.

1 D. Product Category Mapping

We categorize the product name details into product categories, such as rice, egg, chicken meat, beef, vegetable, fruit, seasoning, crackers, drinks, sweets, soap, shampoo, kitchen cleaner, etc. Product categories is very useful to compare product prices across many countries under similar product type. We use OpenAI GPT-5-mini to categorize product.

1 E. Measurement Scale and Quantity Standardization

Products in many countries are not always marketed in simple well know measurement scale such as liter, kilogram, meter, pcs. They could be offered in pounds, lbs, oz, gram, mililiter, inch, dozen, including measurement scale in local language. For example : لتر (=Liter), لغ' (=Kilogram), رطل (=Pounds), ЛИТРА (=Liter), КГ (=Kilogram), リットル (=Liter), キロ(=Kilogram), 公斤 (=Kilogram), 升 (=Liter), กิโลกรัม (=Kilogram), ลิตร (=Liter) . They are also not always marketed in simple exact 1 unit quantity, instead in 900 ml, 2 pounds, 2 x 5 gram , 10 x 1.5 kilogram, and so on.

Therefore, we standardize the quantity and measurement scale to be in Kilogram/ Liter/ Meter/ Pcs/ Carton unit. We use custom Python function code for this. Using any GPT model is not recommended for this standardization because it increase time significantly on agentic workflow, and also cost. Using custom Python function code takes much shorter time, but we need to update the function code for every new unrecognized quantity and measurement scale.

1 F. Embed Price Currency Conversion

Product marketed in local market is not always using USD currency, but using its national currency. For example, Japan's national currency is JPY/Yen. Saudi Arabia's national currency is SAR/Riyal. Most European countries' national currency is EUR/Euro.

However for price comparison across multiple countries, we have to use one single currency, so beef price from multiple countries have to be compared in USD, or in AUD, or in CNY, for example.

We use ExchangeRate-API to download the latest currency conversion every day. Then we use custom Python function code to convert local price into USD/EUR/CHF/JPY/CNY/SGD/AUD for price comparison in single currency.

Using OpenAI GPT models directly to convert price currency is not recommended, because it increase time significantly in our agentic workflow, and also cost.

1 G. Final Tabular Data Insertion to Supabase

The agentic workflow store the end data into Supabase in secret schema, which is not exposed to public/anon. We set up "end_data" schema which is exposed to public/anon for front end Maintelyd website. We set up also CRON job to refresh the end_data schema twice per day using data from secret schema.

2. Product Price Topic Requests

We prepared list of product price topic, to be executed by the agentic workflow. Here are example of product price topic :

  1. web search on specific topics : Quanto custa 1 kg de ovo de galinha fresca no Brasil? ; Wie viel kostet 1 L Sojaöl in Deutschland ? ; Hoeveel kos 1 kg vars rooi koejawels in Suid-Afrika? ; كم سعر 1 كغ من البيض الطازج في السعودية؟ ; Сколько стоят 1 кг моркови в России? ; 日本では、米5kgはいくらですか？; 在中国，新鲜牛肉1公斤多少钱？; มะนาวเขียว 1 กิโลกรัม ราคาเท่าไหร่ในประเทศไทย? ; How much does 1 kg fresh avocado cost in Singapore ?
  2. deep data extraction on targeted URLs : we don't send any product price topic, but URL links to be extracted, for example https://supermarket.yogyaonline.co.id/supermarket/minuman-air-mineral/category ; https://www.sayurbox.com/category/premium-1-c45106cf/sub-category/premium-buahsegar-1-0287a9e7?selectedCategoryType=ops&touch_point=navigation%20bar&origin=null%3A%2F%2Fwww.sayurbox.com&sortBy=most_relevant ; https://www.blibli.com/cari/ikan%20segar%20per%201%20kg
  3. Google Shopping API : similar as web search topics , without asking and ? . For example : 1 kg de queijo no Brasil ; 1 kg de farine en France ; 1 كغ لحم دجاج طازج ; 新鮮なカカオ豆1kg

So product price topic are using country's native language, with purpose to get localized product price results in that country.

A total 3100 product price topics + URLs are listed in Supabase, to be executed by the agentic workflow. We tried cover as many topics as possible, while currently the topics mostly about raw foods/agricultural - garden - dairy - fisheries products. With more budget, we will expand the product price topic lists and URL lists.

3. Automation Pipeline

Here are simplified automation pipeline from product price discovery until end-data storage :

  1. One product price topics/URL is feed to agentic workflow.
  2. The agentic workflow process it , as explained in section 1 from 1A until 1G.
  3. That product price topic/URL marked as done.
  4. The pipeline stop for 180 minutes.
  5. Then next product price topics/URL is feed to agentic workflow. Begin again from 1-5. And so on.

We run automation in Railway platform.

The pipeline could be run for every less than 180 minutes. Even that is better so more data are discovered in one day. However, we have budget limitation that we have to limit maximum OpenAI API cost to be 0.6 USD per day. We have limited our monthly budget OpenAI API in 18 USD. Railway monthly cost is 5 USD. Supabase cost is still free. So total monthly budget is 23 USD. This Maintelyd project is still fully self-funded.

4. Front End Maintelyd Website

We developed the web front end with TypeScript framework in Vercel platform. We purchased maintelyd domain in Cloudflare with cost 10.46 USD.

5. Project Timeline

Before Open AI Build Week 13 - 21 July 2026

January - February 2026 : built and test the agentic workflow from scratch in Jupyter Notebook. Learning from LangGraph Academy. With small support from ChatGPT. March - April 2026 : finishing the agentic workflow in VS Code. Built automation pipeline. Prepared product price topics and URLs. With large support from ChatGPT. May 2026 : Setup GitHub, Railway, Supabase. Launch Maintelyd website. Fixing minor bugs. With large support from ChatGPT. June 2026 - Now : Monitoring project cost, Maintelyd LinkedIn. Occasionally do data validation.

During Open AI Build Week 13 - 21 July 2026

We truly never used Codex before. But actually we are quite overwhelmed by this old practice : "ChatGPT recommends code changes ---> We find the problematic code lines (guided by ChatGPT), then apply that change in VS Code ---> Copy paste full code to GitHub repo ---> Commit changes". After we know that Codex could apply code change directly to GitHub, we give it a try and finally solve these 2 problems :

a. bad UI appearance in mobile browser view, chart are "compressed" by large filters. On desktop browser view, the huge tooltip shadowed the chart dots and line.

b. charts and table are rigid. Meanwhile in many occasions user wants to ask customized product price analysis, and the best feature for this purpose is front end agent. The database are ready, but at that time front end agent was still not available. We are using Codex specifically starting from 18 July 2026, after we received Devpost email in 18 July early morning about this OpenAI Build Week.

Challenges we ran into (focus on during OpenAI Build Week period)

Solved Problem

a. bad UI appearance in mobile browser view, chart are "compressed" by large filters. On desktop browser view, the huge tooltip shadowed the chart dots and line. I taught Codex the GitHub product-price-intelligence repo structure. Meanwhile we didn't taught Codex about deep page.tsx and dashboard.tsx TypeScript code structure (we really don't have this knowledge, those were previously from ChatGPT). But Codex intelligently acknowledged folder structure, edited the TypeScript code, by creating additional code for mobile view specific, and edit the tooltip from long vertical shape to be n row x 3 columns.

b. No front end agent on Maintelyd website, where user could ask customized product price analysis. The first version agent that Codex built was not "an agent", but merely just reporting templates even with many weird product categories. That feature also was repeatedly created by Codex. This is because Codex was still scary of SQL code generated that could update/insert/delete data wrongly on Supabase table. After we ensured that only Read access was given to the public/anon role in Supabase RLS policy, then finally the Codex build the real agent :-)))) . Even in the instruction , Codex has written to prevent any SQL code that Update/Write/Delete data.

Unsolved Problem

a. GPT-5.6 cost. We found that with GPT-5.6, our 1x run agentic workflow for web search cost jump into 0.5 USD. Meanwhile the current cost with GPT-5.4 is just 0.08 USD. With GPT-5.6, the daily cost would be 4 USD and monthly budget would be 120 USD, which is already exceeding our monthly budget at 23 USD.

b. Our limited monthly budget, and agentic workflow idea novelty. Our Maintelyd project is still self-funded. And we are still not sure if this product price discoveries project is novel enough. We might believe that many people out there could have similar data discoveries project as Maintelyd.

Accomplishments that we're proud of

a. From ChatGPT and OpenAI GPT models, we could have live portfolio, which is website + live data that keep coming discovered every day. In the past, this kind of project would have needed multiple specialists (backend developer and front end developer) and also expensive cost to hire them.

b. By help from Codex, we finally able to build front end agent, which consuming our ready database. This agent could understand human language and do SQL coding dynamically to finally answer user's custom price analysis. Agent is the next gen of data visualization. We keep charts and tables as for regular monitoring and reporting. But for custom data analysis , we cannot always rely manually to our data analyst team. Instead, we ask agent to do that. Our front end agent in Maintelyd, Product Price Master Agent, for now capable to generate analysis result in table form. Next, we will add capabilities to create charts to it !

c. Two birds one stone. From point b, we also discover "incremental knowledge agent". Because our front end agent is exposed to the database that always updated every day, with new price data that keep coming every day. The agent is able to give answer about the latest price trend, and also the past historical price trend. No need manual training setup periodically, the agent is already exposed to the regularly-refreshed database !

d. Although unfortunately we could not continue utilizing GPT-5.6 due to cost constraint explained above, our Maintelyd project is fully powered by OpenAI. The agentic workflows, automation pipeline, frontend website were developed by ChatGPT. The website UI enhancement and front end agent were developed by Codex during this OpenAI Build Week Submission Period. And models inside agentic workflows and inside front end agent are 100 % OpenAI (GPT-5.4 and GPT-5-mini). No gemini, claude, deepseek etc models inside this Maintelyd project. I don't want to lie that I utilize GPT-5.6 in the demo video, so this point d is very important ground truth.

What we learned specifically on ChatGPT and Codex

a. For building complex apps/system that has multiple parts, it is better to develop part-by-part. So ask ChatGPT/Codex about first part until finish. Then go to next part.

b. For troubleshooting, provide screenshot pinpoint the problem, the problem context (the problem xxx, while it should be yyy), optionally additional clue that is helpful for agent for starting point troubleshooting.

c. If the Codex doesn't implement what you want (and annoyingly repeatedly build features you don't want), in one prompt it's ok to express your disappointment and resentment to the Codex . Then in the same prompt, in calm context put additionally why you keep wanting your specific objective and safety assurance to the Codex. What I mean safety assurance : that this implementation will not cause any bad impact , because you have ensured it (explain that also).

What's next for Maintelyd

a. If there is an opportunity for additional budget, we will set the automation in more frequent agentic workflow, less than currently every 180 minutes. So more data could be discovered and then global product price become more meaningful.

b. Make additional product price topics to be discovered. For example : personal health care, house/apartment price, transportation cost, energy cost.

c. Implement the second 2nd use case , which could be social sentiment. This social sentiment insights are really useful : to match people interest with correct product/services (for better targeted marketing), to predict/forecast the next outcome after specific insights occur. GPT-5.6 become more useful for this 2nd use case. My initial idea for next GPT-5.6 utilization could be : equipped with MCP instagram, X, tiktok (this MCP is to do posts sampling against every selected topics inside those 3 platforms periodically), and equipped with skills (to quantify the positive and negative statements, interactivity strength, inside the posts), the agent analyze the relative interest power real time. Movement of interest power from one topic to another topics are worth noting for social sentiment insights. Not only sets of topics to be analyzed, but the agent could detect new emerging topics. The results are stored again in tabular database. Then user through front end agent ask anything to the database, about important applicable insights that are helpful to their marketing questions/audience targeting questions/etc.

Built With

Share this project:

Updates