Home Page
Job Search With Global Search bar
Default Job List View
Job Description From Hacker News
Job Analytics
Application Architecture
HackerX Intro
Newsletter Signup
Newsletter

Inspiration

The inspiration behind "Hacker Jobs" stems from the desire to solve a common problem faced by job seekers—the challenge of finding suitable tech and hacker community-related job opportunities efficiently. We recognized that the Hacker News platform contains a wealth of job listings in the tech industry. Leveraging this data, the goal was to create a platform that offers job seekers a centralized and user-friendly interface to discover relevant job listings quickly.

Additionally, we wanted to go beyond conventional keyword-based job searches and enhance the job-matching process. By utilizing the OpenAI API, the project sought to extract valuable metadata from job descriptions. This metadata includes key information like required skills, job roles, and company details. Integrating this natural language processing capability into the platform aims to provide users with more accurate and relevant job matches, helping them find their dream jobs effectively.

What it does

Hacker Jobs serves as an all-inclusive platform catering to both job seekers and job posters. For job seekers, the platform provides a streamlined and intuitive interface to browse and search for tech and hacker community-related job opportunities. The platform gathers historical and real-time data from the Hacker News platform using the Hacker News API. Through data analysis and matching algorithms, "Hacker Jobs" ensures that job seekers are presented with relevant job listings that align with their skills and interests.

To enhance the accuracy of job matching, "Hacker Jobs" leverages the power of the OpenAI API. By extracting metadata from job descriptions, the platform gains deeper insights into the job roles, required qualifications, and other important details. This metadata-driven approach enables the platform to provide job seekers with highly personalized and suitable job recommendations, ultimately leading to a more efficient job search process.

For job posters, "Hacker Jobs" offers an easy-to-use and efficient platform to post job listings. Employers and recruiters can input job details, qualifications, and other relevant information seamlessly, ensuring their job listings reach the right audience of qualified and motivated individuals within the tech and hacker community.

The platform also provides a way to analyze historical data using the Metabase open-source project. This will help job seekers to analyze and understand the market trends.

How we built it

The development of "Hacker Jobs" involves a robust tech stack that optimizes cost, performance, and scalability:

Database - TiDB Cloud and TiDB Serverless: The core of the platform's database infrastructure relies on TiDB Cloud and TiDB Serverless. TiDB Cloud provides a distributed SQL database that handles the storage and retrieval of vast amounts of data efficiently. TiDB Serverless complements this by dynamically scaling resources based on demand, reducing costs during periods of lower activity.

Backend - AWS Lambda: The backend of "Hacker Jobs" is implemented using AWS Lambda, a serverless compute service. AWS Lambda automatically scales resources to match API request loads, ensuring the platform can efficiently handle varying levels of user activity. The integration of AWS Lambda with TiDB Cloud and TiDB Serverless facilitates seamless data access and management, forming a robust foundation for the platform's functionalities.

Frontend - Next.js and Vercel: The frontend of "Hacker Jobs" is built using Next.js, a popular React framework known for its performance and ease of development. Next.js enables the creation of user-friendly interfaces that are responsive and interactive. The frontend is hosted on Vercel, a deployment platform that streamlines hosting and automates various aspects of the development pipeline. This allows the team to focus on core development tasks without worrying about server management.

Analytics - Metabase on AWS EC2: In addition to job matching and search functionalities, "Hacker Jobs" provides valuable insights and analytics to users. This is made possible by integrating the open-source project Metabase. Metabase is deployed on an AWS EC2 instance, offering scalability and processing power for generating insightful reports and visualizations based on the data collected from the platform.

Supporting Services - CloudWatch, Secrets Manager: To ensure the platform's performance and stability, "Hacker Jobs" relies on AWS CloudWatch for monitoring and logging various aspects of the system. CloudWatch provides valuable insights into the platform's health, which helps the team identify and address potential issues promptly. Secrets Manager plays a vital role in securely managing sensitive data, such as API keys and credentials, ensuring the protection of critical information.

Challenges we ran into

Throughout the development process, the team encountered several notable challenges:

Throttling and Data Retrieval: Dealing with the vast volume of data available on the Hacker News platform presented challenges related to throttling and efficient data retrieval. We optimize API requests and implement strategies to handle the data retrieval process effectively.

Metadata Extraction from Job Descriptions: Extracting meaningful metadata from job descriptions in an unstructured format proved to be a significant challenge. To overcome this, the team integrated the OpenAI API, which required fine-tuning and careful handling to ensure accurate and relevant metadata extraction.

Integration of Metabase: Implementing Metabase on AWS EC2 for serving analytics posed challenges related to deployment, configuration, and integration with the platform's data sources. The team had to ensure a smooth and seamless flow of data from the platform to Metabase for generating insightful reports and visualizations.

Accomplishments that we're proud of

Despite the challenges faced, the team achieved several noteworthy accomplishments:

Optimization of Data Retrieval: The team successfully optimized API requests to Hacker News, allowing for efficient data retrieval without running into throttling issues. This optimization ensured a smooth experience for users searching for job listings.

Metadata-Driven Job Matching: By integrating the OpenAI API, the team accomplished the extraction of valuable metadata from job descriptions. This metadata-driven approach significantly improved the accuracy of job matching, providing users with more relevant and personalized job recommendations.

Analytics and Insights: The integration of Metabase on AWS EC2 enabled "Hacker Jobs" to offer comprehensive analytics and insights to users. Providing valuable data-driven reports and visualizations, the platform empowers both job seekers and job posters with deeper market insights.

What we learned

Throughout the development journey, the team gained valuable learning experiences:

Effective API Integration: The team honed their skills in integrating external APIs like Hacker News API and the OpenAI API. This allowed them to leverage external data and services to enhance the platform's capabilities.

Cost-Optimization with Serverless: By utilizing AWS Lambda and TiDB Serverless, the team learned to architect cost-efficient solutions that dynamically scale resources based on demand, thereby reducing unnecessary expenses.

Open-Source Analytics Integration: The integration of Metabase on AWS EC2 provided valuable insights into implementing and managing open-source analytics tools. The team acquired knowledge in generating data visualizations and reports for users' benefit.

What's next for Hacker Jobs ( Completed As part of Final Submission Enhancement )

The future roadmap for "Hacker Jobs" includes continuous enhancements and additional features to further benefit job seekers and job posters:

Resume Upload and Filtering: A major planned enhancement is allowing job seekers to upload their resumes. By analyzing uploaded resumes and extracting qualifications, "Hacker Jobs" can implement a filtering mechanism that matches job seekers to job listings more accurately.

Final Submission Enhancement

As part of the final submission enhancement, we are thrilled to introduce HackerX. By subscribing to HackerX, you'll gain access to a treasure trove of valuable information tailored specifically to your skills, interests, and career goals. Here's what you can expect from our monthly newsletters.

How Does It Work?

Signing up for HackerX's monthly newsletter is a breeze:

Newsletter Subscription: Simply visit the "HackerX" section on our platform and subscribe to the newsletter. During the process, you'll have the option to upload your resume, which helps us personalize the content you receive.
Resume Analysis: Our advanced natural language processing (NLP) algorithms will carefully analyze the details in your resume. Rest assured that your data is treated with the utmost confidentiality and security.
Tailored Content: Once our system processes your resume, you'll start receiving your personalized monthly newsletter. We take pride in delivering content that is relevant and valuable to you.
Customize Your Preferences: We understand that your career goals may evolve over time. With HackerX, you can easily update your preferences and fine-tune the content you receive, ensuring it stays aligned with your aspirations.

The resume is processed effectively with the help of the AWS Textract to extract the required fields like tech stacks, company, experience, name and etc.,