Technical Terms You need to Know Before stepping into Big Data Processing

 

List Of Technical Terms 

Cluster : 

group of computers or servers.

Parallel processing : 

group of computers or servers used to do same task.

 Like Multiple developers hired to develop same application by splitting their tasks. Similar to that if we have 320 MB data to process and have 5 machines, Hadoop will split it into five 64 MB chunks then five parallelly processes by 5 machines. It  will reduce 4 times processing time.

Concurrency : 

Single thread or process will handle multiple works or tasks in a time period.
Example :

Technically – a process got task to retrieve 100 accounts data through API. API will accept only one account per request, process should make 100 API call to complete that task.  Normally, a process will call API for account1 and wait ideally until get response. In concurrent scenario, meanwhile it waiting for Api response for Account1, it will send requests for other accounts also to API. Suppose, process took 2 milleseconds to send request and API 500 milliseconds to respond, instead ideal for 500 milliseconds, process can send another 250 requests to API. In our case, it will request for remaining 99 accounts and overall task will complete in 1 second if go with concurrecncy but ordinary process will 50 seconds to complete same task.

Lay man – Took scenario as a girl preparing lunch. She prepared things to cook rice, put those in cooker on stove and waiting for rice to cook in normal. But In concurrency mode, she will peel, slice, chop, dice the vegetable and prepare other ingredients to cook curry, samabar, rasam  or gravy kind of multitasking.

Node: 

Node = a computer or server

Worker : 

worker = slave  a computer or server or node which will follow the Master instructions.

Master : 

a node which is manager of worker or slave nodes. It will guide to worker nodes what to do, how to do the given task, handle worker node failures, maintain the meta data jobs and tasks, monitor worker nodes, maintain data redundancy and responsible for  job completion.

Distributed computing processing:

 nothing parallel processing. But architecture may differ and network may be intra or inter. In actual case, company will use intra networked cluster for security reason, now cloud era came. Hadoop or spark will follow Mater – slave architecture.

Batch Processing : 

processing a volume of data by splitting into chunks in predefined size which are readily available to process.

Example :  you have 10 records to process, you predefined batch size as 5. Then it will process top 5 records first and will process next 5 after completing first 5. In distributed or parallel processing case, first will be processed by one node and next 5 by another node.
Micro Batch processing : will process even with very small number of records.

Streaming : 

is a real time processing. Even for single request or record, a independent tread will launch to complete that task for that record as soon as possible.
Example : Bank transaction, order placing, content delivery apps like face book, you tube, Instagram and messaging, live telecast etc……

Redundancy :

 Storing multiple copies of same data in different machines to prevent data loss.

Retention Period : 

Data will be stored for certain period of predefined time and After that , that data will be deleted. Use - To avoid storing sensitive data for long time, use storage efficiently etc...