Skip to content

manoelbritto/Data_Engineering_PostgreSQL

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Project: Job Positions data set

This project implemented:

  • PostgreSQL Database,
  • Data modeling (ERD),
  • Data Engineering (import CSV files)
  • Queries (SQL) to understand the data

Data Modeling

Six tables created with one-to-many relationships. Two of them, Dept_Emp and Dept_Manager, can be considered as bridge tables.

This is the ERD created:

GitHub Logo

Data Engineering

After ERD created, a script with create table and drop table were generated to be applied in the PostgreSQL server. These scripts you can find in ‘ScriptDB’ folder as soon as many other queries and statements generated, which you can see in the ‘QueryDB.sql’ in the same ‘ScriptDB’ folder.

The CSV data contents were imported using pgAdmin. The content files can be found inside ‘data’ folder.

Data Analysis

Inside of ‘Analysis’ folder, there is a jupyter notebook(.ipynb) file that has an analysis using a bar chart to show the average salaries per job position.

GitHub Logo

SQL query

Besides, I created a folder 'ScriptDB' which has a file called QueryDB.sql. With these queries, I could have a better overview about the data.

Prerequisites

To run this project, you should have a PostgreSQL server installed in your machine, if you don’t have to go to this link https://www.postgresql.org/download/ and installed it.

And a Jupyter notebook https://www.anaconda.com/distribution/.

Install

To install sqlalchemy and psycopg2 to run in your Jupyter notebook, you have to use these prompt commands:

*conda install -c anaconda sqlalchemy *conda install -c anaconda sqlite *pip install psycopg2

Features

  • Modeling Entity Relationship Diagram (ERD) with visual Paradigm,
  • Langues: Python and SQL

Libraries:

  • Pandas,
  • sqlalchemy
  • psycopg2
  • matplotlib

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published