3.HDFS Overview

Hadoop File System was developed using distributed file system design. It is run on commodity hardware. Unlike other distributed systems, HDFS is highly faulttolerant and designed using low-cost hardware.

HDFS holds very large amount of data and provides easier access. To store such huge data, the files are stored across multiple machines. These files are stored in redundant fashion to rescue the system from possible data losses in case of failure. HDFS also makes applications available to parallel processing.

Features of HDFS

• It is suitable for the distributed storage and processing.
• Hadoop provides a command interface to interact with HDFS.
• The built-in servers of namenode and datanode help users to easily check the status of cluster.
• Streaming access to file system data.
• HDFS provides file permissions and authentication.

HDFS Architecture

Given below is the architecture of a Hadoop File System.

HDFS follows the master-slave architecture and it has the following elements.

Namenode

The namenode is the commodity hardware that contains the GNU/Linux operating system and the namenode software. It is a software that can be run on commodity hardware. The system having the namenode acts as the master server and it does the following tasks:

• Manages the file system namespace.
• Regulates client’s access to files.
• It also executes file system operations such as renaming, closing, and opening files and directories.

Datanode

The datanode is a commodity hardware having the GNU/Linux operating system and datanode software. For every node (Commodity hardware/System) in a cluster, there will be a datanode. These nodes manage the data storage of their system.

• Datanodes perform read-write operations on the file systems, as per client request.
• They also perform operations such as block creation, deletion, and replication according to the instructions of the namenode.

Block

Generally the user data is stored in the files of HDFS. The file in a file system will be divided into one or more segments and/or stored in individual data nodes. These file segments are called as blocks. In other words, the minimum amount of data that HDFS can read or write is called a Block. The default block size is 64MB, but it can be increased as per the need to change in HDFS configuration.

Goals of HDFS

• Fault detection and recovery : Since HDFS includes a large number of commodity hardware, failure of components is frequent. Therefore HDFS should have mechanisms for quick and automatic fault detection and recovery.
• Huge datasets : HDFS should have hundreds of nodes per cluster to manage the applications having huge datasets.
• Hardware at data : A requested task can be done efficiently, when the computation takes place near the data. Especially where huge datasets are involved, it reduces the network traffic and increases the throughput.

1.Introduction to Hadoop

Hadoop is an Apache open source framework written in java that allows distributed processing of large datasets across clusters of computers using simple programming models. A Hadoop frame-worked application works in an environment that provides distributed storage and computation across clusters of computers. Hadoop is designed to scale up from single server to thousands of machines, each offering local computation and storage.

Hadoop Architecture

Hadoop framework includes following four modules:

• Hadoop Common: These are Java libraries and utilities required by other Hadoop modules. These libraries provides filesystem and OS level abstractions and contains the necessary Java files and scripts required to start Hadoop.
• Hadoop YARN: This is a framework for job scheduling and cluster resource management.
• Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.
• Hadoop MapReduce: This is YARN-based system for parallel processing of large data sets.

We can use following diagram to depict these four components available in Hadoop framework.

Since 2012, the term "Hadoop" often refers not just to the base modules mentioned above but also to the collection of additional software packages that can be installed on top of or alongside Hadoop, such as Apache Pig, Apache Hive, Apache HBase, Apache Spark etc.

MapReduce

Hadoop MapReduce is a software framework for easily writing applications which process big amounts of data in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.

The term MapReduce actually refers to the following two different tasks that Hadoop programs perform:

• The Map Task: This is the first task, which takes input data and converts it into a set of data, where individual elements are broken down into tuples (key/value pairs).
• The Reduce Task: This task takes the output from a map task as input and combines those data tuples into a smaller set of tuples. The reduce task is always performed after the map task.

Typically both the input and the output are stored in a file-system. The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks.

The MapReduce framework consists of a single master JobTracker and one slaveTaskTracker per cluster-node. The master is responsible for resource management, tracking resource consumption/availability and scheduling the jobs component tasks on the slaves, monitoring them and re-executing the failed tasks. The slaves TaskTracker execute the tasks as directed by the master and provide task-status information to the master periodically.

The JobTracker is a single point of failure for the Hadoop MapReduce service which means if JobTracker goes down, all running jobs are halted.

Hadoop Distributed File System

Hadoop can work directly with any mountable distributed file system such as Local FS, HFTP FS, S3 FS, and others, but the most common file system used by Hadoop is the Hadoop Distributed File System (HDFS).

The Hadoop Distributed File System (HDFS) is based on the Google File System (GFS) and provides a distributed file system that is designed to run on large clusters (thousands of computers) of small computer machines in a reliable, fault-tolerant manner.

HDFS uses a master/slave architecture where master consists of a single NameNode that manages the file system metadata and one or more slave DataNodes that store the actual data.

A file in an HDFS namespace is split into several blocks and those blocks are stored in a set of DataNodes. The NameNode determines the mapping of blocks to the DataNodes. The DataNodes takes care of read and write operation with the file system. They also take care of block creation, deletion and replication based on instruction given by NameNode.

HDFS provides a shell like any other file system and a list of commands are available to interact with the file system. These shell commands will be covered in a separate chapter along with appropriate examples.

How Does Hadoop Work?

Stage 1

A user/application can submit a job to the Hadoop (a hadoop job client) for required process by specifying the following items:

1. The location of the input and output files in the distributed file system.
2. The java classes in the form of jar file containing the implementation of map and reduce functions.
3. The job configuration by setting different parameters specific to the job.

Stage 2

The Hadoop job client then submits the job (jar/executable etc) and configuration to the JobTracker which then assumes the responsibility of distributing the software/configuration to the slaves, scheduling tasks and monitoring them, providing status and diagnostic information to the job-client.

Stage 3

The TaskTrackers on different nodes execute the task as per MapReduce implementation and output of the reduce function is stored into the output files on the file system.

Advantages of Hadoop

• Hadoop framework allows the user to quickly write and test distributed systems. It is efficient, and it automatic distributes the data and work across the machines and in turn, utilizes the underlying parallelism of the CPU cores.
• Hadoop does not rely on hardware to provide fault-tolerance and high availability (FTHA), rather Hadoop library itself has been designed to detect and handle failures at the application layer.
• Servers can be added or removed from the cluster dynamically and Hadoop continues to operate without interruption.
• Another big advantage of Hadoop is that apart from being open source, it is compatible on all the platforms since it is Java based.

What happens if a archive log destination is filled up in oracle

This is a common problem Oracle DBA’s usually face.

When our Oracle database running in archive log mode with automatic archiving turned on. And When the archive
log destination disk space becomes full, the automatic archiving will stop.

When this happens we will get errors like ORA-00257

ORA-00257: archiver error. Connect internal only, until freed.

Cause: The archiver process received an error while trying to archive a redo log.
If the problem is not resolved soon, the database will stop executing transactions.
The most likely cause of this message is the destination device is out of space to store the redo log file.

Action: Check archiver trace file for a detailed description of the problem.
Also verify that the device specified in the initialization parameter ARCHIVE_LOG_DEST is set up properly for
archiving.

When all of the online redo logs are filled up, the database will be in a hung state.

If we attempt to manually archive the files with commands “archive log next” or “archive log all”
you may receive errors

solutions :-

1. Increase the filesyatem where the archive logs are been generated.

If it is a veritas filesyatem we can resize in run time.

The commands are.

Check how much space is available in diskgroup.

vxassist — create, relayout, convert, mirror, backup, grow, shrink, delete, and move volumes

maxsize – Reports the maximum size a volume could be created with given the attributes specified and
given the free storage available.

e.g -

root@datahouse1# vxassist -g lduprd_dg maxsize
Maximum volume size: 182728704 (89223Mb)

Resize the volume.

The example below resizes a filesystem to 8 GB.

root@datahouse1# /usr/lib/vxvm/bin/vxresize -g lduprd_dg lduprd_db231_lv +8g

2. Delete all the archivelogs in the archive destination.

Then connect to recovery catalog and target database and issue the command in rman prompt.

allocate channel for maintenance type disk;
change archivelog all crosscheck;
release channel;

Note :- Crosscheck command checks whether the specified backups and copies exist. If RMAN cannot find
backup pieces, it marks them as having the status expired. It marks all other types of absent files–image
copies and archived redo logs–as deleted.

If the files are on disk, RMAN queries the file headers. For other device types, RMAN queries the media
manager to see whether the file exists in the media management catalog.

Take a incremental level 0 database backup from rman.

3. Take the backup of all archivelogs through RMAN

Backup archivelog all delete input.

4. Use the command:

alter system archive log all to ‘/u07/app/oradata/DBADB/archive’;

The command archives all redo log file groups that are full but have not been archived.

The gives Oracle a valid location for the archive logs. Even after using this the archive log destination
parameter is still invalid and automatic achive does not work. This solution is best for systems
that do not archive regularly but cannot be shutdown. You can also use this to allow you to do a
Shutdown immediate instead of Shutdown abort.

Shutdown and restart of the database resets the archive log destinstation parameter to be valid.
Do not forget to make disk space available before starting the database.

5. Use the REOPEN attribute of the LOG_ARCHIVE_DEST_n init<SID>.ora parameter.

LOG_ARCHIVE_DEST_n=’<path> REOPEN=30′

Or run the following comand:

alter system set LOG_ARCHIVE_DEST_n = ‘reopen’

The REOPEN and the NOREOPEN attributes of the LOG_ARCHIVE_DEST_n parameter specify the minimum number
of seconds before the archiver processes (ARCn) or the log writer process (LGWR) should try again to
access a previously failed destination. You can turn off the attribute by specifying NOREOPEN.

6.Use the commands:

alter system archive log stop;
alter system archive log start;

Super fast Database Copying/Cloning

A database cloning procedure is especially useful for the DBA who wants to give his developers a full-sized TEST and DEV instance by cloning the PROD instance into the development server areas.

This Oracle clone procedure can be use to quickly migrate a system from one UNIX server to another. It clones the Oracle database and this Oracle cloning procedures is often the fastest way to copy a Oracle database.

STEP 1: On the old system, go into SQL*Plus, sign on as SYSDBA and issue: “alter database backup controlfile to trace”. This will put the create database syntax in the trace file directory. The trace keyword tells oracle to generate a script containing a create controlfile command and store it in the trace directory identified in the user_dump_dest parameter of the init.ora file. It will look something like this:

STARTUP NOMOUNT
CREATE CONTROLFILE REUSE DATABASE "OLDLSQ" NORESETLOGS
NOARCHIVELOG
MAXLOGFILES 16
MAXLOGMEMBERS 2
MAXDATAFILES 240
MAXINSTANCES 1
MAXLOGHISTORY 113
LOGFILE
GROUP 1 ('/u03/oradata/oldlsq/log1a.dbf',
'/u03/oradata/olslsq/log1b.dbf') SIZE 30M,
GROUP 2 ('/u04/oradata/oldlsq/log2a.dbf',
'/u04/oradata/oldlsq/log2b.dbf') SIZE 30M
DATAFILE
'/u01/oradata/oldlsq/system01.dbf',
'/u01/oradata/oldlsq/mydatabase.dbf'
;
# Recovery is required if any of the datafiles are restored
# backups, or if the last shutdown was not normal or immediate.
RECOVER DATABASE
# Database can now be opened normally.
ALTER DATABASE OPEN;


STEP 2: Shutdown the old database

STEP 3: Copy all data files into the new directories on the new server. You may change the file names if you want, but you must edit the controlfile to reflect the new data files names on the new server.

rcp /u01/oradata/oldlsq/* newhost:/u01/oradata/newlsq
rcp /u01/oradata/oldlsq/* newhost:/u01/oradata/newlsq
rcp /u03/oradata/oldlsq/* newhost:/u03/oradata/newlsq
rcp /u04/oradata/oldlsq/* newhost:/u04/oradata/newlsq


STEP 4: Copy and Edit the Control file – Using the output syntax from STEP 1, modify the controlfile creation script by changing the following:

Old:

CREATE CONTROLFILE REUSE DATABASE "OLDLSQ" NORESETLOGS

New:

CREATE CONTROLFILE SET DATABASE "NEWLSQ" RESETLOGS

STEP 5: Remove the “recover database” and “alter database open” syntax

# Recovery is required if any of the datafiles are restored
# backups, or if the last shutdown was not normal or immediate.
RECOVER DATABASE
# Database can now be opened normally.
ALTER DATABASE OPEN;


STEP 6: Re-names of the data files names that have changed.

Save as db_create_controlfile.sql.

Old:

DATAFILE
'/u01/oradata/oldlsq/system01.dbf',
'/u01/oradata/oldlsq/mydatabase.dbf'

New:

DATAFILE
'/u01/oradata/newlsq/system01.dbf',
'/u01/oradata/newlsq/mydatabase.dbf'

STEP 7: Create the bdump, udump and cdump directories

cd $DBA/admin
mkdir newlsq
cd newlsq
mkdir bdump
mkdir udump
mkdir cdump
mkdir pfile


STEP 8: Copy-over the old init.ora file

rcp $DBA/admin/olslsq/pfile/*.ora newhost:/u01/oracle/admin/newlsq/pfile

STEP 9: Start the new database

@db_create_controlfile.sql

STEP 10: Place the new database in archivelog mode

Database Cloning Using Hot Backup

Below steps helps you in performing database cloning using hot backup

Assumptions:

1. directory structure is different in both source and target servers
2. Oracle version : 10.2.0.5
3. OS version : Linux 5
4. target database name is same as source database name
5. DATABASE SHOULD BE IN ARCHIVEMODE

step 1 : Take the hot backup of source database

If the souce database in not in Archivemode, alter database to archivemode.

SQL> STARTUP MOUNT;

SQL> ALTER DATABASE ARCHIVELOG;

SQL> ALTER DATABASE OPEN;

sql> alter database begin backup;

$ copy datafiles to backup location

sql> alter database end backup;

step 2 : Take controlfile trace and pfile or spfile (that was using by the source database)

step 3 : Install Oracle software on another machine (choose “Install only” option in OUI). Don’t create any database

step 4 : Copy all the files (including trace file and pfile or spfile) from source server to target server either using FTP or rcp
Note: Not necessary in copying control files and redologfiles

step 5 : Place pfile or spfile in “dbs” directory on target

step 6 : Copy the remaining files to their respective locations (If any directories are missing, do create them)

step 7 : Connect as sysdba and Startup the database in nomount stage

step 8 : Edit the trace file (that was copied) and generate a create controlfile script from it.
Modify the script and specify the new locations of the files.

step 9 : Execute the controlfile script which will create controlfiles in the location specified in CONTROL_FILES parameter.
Once control files are created, database will be forwarded to MOUNT state.

sql> @create_controlfile.sql

step 10 : Finally, Open the database with resetlogs option

sql> startup nomount

sql> alter database mount;

sql> recover database using backup controlfile until cancel;

sql> alter database open resetlogs;

REPLICATION

Replication is the process of copying and maintaining database objects in multiple databases that make up a distributed database system. Replication can improve the performance and protect the availability of applications because alternate data access options exist. For example, an application might normally access a local database rather than a remote server to minimize network traffic and achieve maximum performance. Furthermore, the application can continue to function if the local server experiences a failure, but other servers with replicated data remain accessible.

Oracle supports two different forms of replication: basic and advanced replication:

1- Basic Replication

With basic replication, data replicas provide read-only access to the table data that originates from a primary (master) site. Applications can query data from local data replicas to avoid network access regardless of network availability. However, applications throughout the system must access data at the primary site when updates are necessary.

2- Advanced (Symmetric) Replication

The Oracle advanced replication features extend the capabilities of basic read-only replication by allowing applications to update table replicas throughout a replicated database system. With advanced replication, data replicas anywhere in the system can provide both read and update access to a table's data. Participating Oracle database servers automatically work to converge the data of all table replicas, and ensure global transaction consistency and data integrity.

Clone Oracle 11g database using cold backup

Prerequisite: U should have valid Cold backup of your all *.dbf Datafiles and control file.

1. Create pfile for new database. Using copy the existing pfile. Using a spfile. SQL> create pfile=’init.ora’ from spfile; Create a new pfile in the $ORACLE_HOME/dbs directory. The new pfile will need to be edited as per new enviornment.

2. Create the controlfile. SQL>alter database backup controlfile to trace as ‘/home/oracle/control.sql’; The .sql file requires editing. Remove all lines from the top of the file up to first STARTUP NOMOUNT line. Remove all lines from the second ‘STARTUP MOUNT’ line till end. Remove any lines that start with — and #. Remove any blank lines. Remove the line ‘RECOVER DATABASE USING BACKUP CONTROLFILE’ Remove the line ‘ALTER DATABASE OPEN RESETLOGS;’ Make a copy of the ‘ALTER TABLESPACE TEMP ADD TEMPFILE ‘/u02/accounts/repodb/oradata/temp01.dbf’ SIZE 524288000 REUSE AUTOEXTEND OFF;’ lines and then remove them from the file. In the ‘CREATE CONTROLFILE’ line the word ‘REUSE’ needs to be changed to ‘SET’. The database name needs changed with new database name. Change the file paths if any. Change the Log mode archivelog/noarchivelog. e.g
STARTUP NOMOUNT
CREATE CONTROLFILE REUSE DATABASE "REPODB" RESETLOGS NOARCHIVELOG
MAXLOGFILES 16
MAXLOGMEMBERS 3
MAXDATAFILES 100
MAXINSTANCES 8 MAXLOGHISTORY 292
LOGFILE
GROUP 1 '/u01/app/oracle/oradata/repodb/redo01.log' SIZE 50M BLOCKSIZE 512,
GROUP 2 '/u01/app/oracle/oradata/repodb/redo02.log' SIZE 50M BLOCKSIZE 512,
GROUP 3 '/u01/app/oracle/oradata/repodb/redo03.log' SIZE 50M BLOCKSIZE 512
DATAFILE
'/u01/app/oracle/oradata/repodb/system01.dbf',
'/u01/app/oracle/oradata/repodb/sysaux01.dbf',
'/u01/app/oracle/oradata/repodb/undotbs01.dbf',
'/u01/app/oracle/oradata/repodb/users01.dbf',
'/u01/app/oracle/oradata/repodb/example01.dbf'
CHARACTER SET WE8MSWIN1252 ;

3. Create the Enviornment File or make new entry in oratab and source the environment. Enviornment File: export ORACLE_SID=repodb export ORACLE_BASE=/u01/app/oracle export ORACLE_HOME=/u01/app/oracle/product/11.2.0 export PATH=$PATH:$ORACLE_HOME/bin oratab: $cd /etc/ vi oratab and make entry in oratab in below format. $ORACLE_SID:$ORACLE_HOME:: Now check if below command shows your database name. $echo $ORACLE_SID

4. Create password file for sqlplus. Use the orapwd utility orapwd file=initrepodb password=*******

5. Start The Database Cloning process. $sqlplus “/ as sysdba” @”/home/oracle/control.sql”; SQL>Control file created.

6. Open Database. SQL>Alter Database open resetlogs;
7. Create temp files. Using the ‘ALTER TABLESPACE TEMP ADD TEMPFILE ‘/u02/accounts/repodb/oradata/temp01.dbf’ SIZE 524288000 REUSE AUTOEXTEND OFF;’ command from step 3.

8. Check Database parameters status Check that the database has opened with the ‘OPEN’ Status. SQL>select status from v$instance; All datafiles Should be ok.i.e Output only be ONLINE and SYSTEM. SQL>select distinct status from v$datafile; Check if alert log are generating.

9. Set new global Database name SQL>alter database rename global_name to ;

10. Create a spfile. SQL>create spfile from pfile;

11. Change the database ID If you are using RMAN then the database ID must be changed. SQL>shutdown immediate; SQL>startup mount; SQL>exit; $nid target=/ Enter ‘Y’. SQL>shutdown immediate SQL>startup mount SQL>alter database open resetlogs

12. Configure TNS Add entries for new database in the listener.ora and tnsnames.ora.

RENAME AN ORACLE DATABASE

Rename an Oracle database using the new dbnewid (also called nid, for new ID) utility:

STEP 1: Backup the database. 

STEP 2: Mount the database after a clean shutdown:

SHUTDOWN IMMEDIATE
STARTUP MOUNT

STEP 3: Invoke the DBNEWID utility (nid) specifying the new DBNAME from the command line using a user with SYSDBA privilege:
nid TARGET=sys/password@TSH1 DBNAME=TSH2
Assuming the validation is successful the utility prompts for confirmation before performing the actions. Typical output may look something like:
C:\oracle\920\bin>nid TARGET=sys/password@TSH1 DBNAME=TSH2
DBNEWID: Release 9.2.0.3.0 - Production
Copyright (c) 1995, 2002, Oracle Corporation. All rights reserved.

Connected to database TSH1 (DBID=1024166118)

Control Files in database:
C:\ORACLE\ORADATA\TSH1\CONTROL01.CTL
C:\ORACLE\ORADATA\TSH1\CONTROL02.CTL
C:\ORACLE\ORADATA\TSH1\CONTROL03.CTL

Change database ID and database name TSH1 to TSH2? (Y/[N]) => Y

Proceeding with operation
Changing database ID from 1024166118 to 1317278975
Changing database name from TSH1 to TSH2
Control File C:\ORACLE\ORADATA\TSH1\CONTROL01.CTL - modified
Control File C:\ORACLE\ORADATA\TSH1\CONTROL02.CTL - modified
Control File C:\ORACLE\ORADATA\TSH1\CONTROL03.CTL - modified
Datafile C:\ORACLE\ORADATA\TSH1\SYSTEM01.DBF - dbid changed, wrote new name
Datafile C:\ORACLE\ORADATA\TSH1\UNDOTBS01.DBF - dbid changed, wrote new name
Datafile C:\ORACLE\ORADATA\TSH1\CWMLITE01.DBF - dbid changed, wrote new name
Control File C:\ORACLE\ORADATA\TSH1\CONTROL01.CTL - dbid changed, wrote new name
Control File C:\ORACLE\ORADATA\TSH1\CONTROL02.CTL - dbid changed, wrote new name
Control File C:\ORACLE\ORADATA\TSH1\CONTROL03.CTL - dbid changed, wrote new name

Database name changed to TSH2.
Modify parameter file and generate a new password file before restarting.
Database ID for database TSH2 changed to 1317278975.
All previous backups and archived redo logs for this database are unusable.
Shut down database and open with RESETLOGS option.
Succesfully changed database name and ID.
DBNEWID - Completed succesfully.

STEP 4: Shutdown the database:

SHUTDOWN IMMEDIATE

STEP 5: Modify the DB_NAME parameter in the initialization parameter file.

The startup will result in an error but proceed anyway.
STARTUP MOUNT
ALTER SYSTEM SET DB_NAME=TSH2 SCOPE=SPFILE;
SHUTDOWN IMMEDIATE

STEP 6: Create a new password file:

orapwd file=c:\oracle\920\database\pwdTSH2.ora password=password entries=10

STEP 7: Rename the SPFILE to match the new DBNAME.

STEP 8: If you are using Windows you must recreate the service so the

correct name and parameter file are used:
oradim -delete -sid TSH1
oradim -new -sid TSH2 -intpwd password -startmode a -pfile
c:\oracle\920\database\spfileTSH2.ora
If you are using UNIX/Linux simply reset the ORACLE_SID environment variable:

ORACLE_SID=TSH2; export ORACLE_SID

STEP 9: Alter the listener.ora and tnsnames.ora setting to match the new

database name and restart the listener:

lsnrctl reload

STEP 10: Open the database with RESETLOGS:

STARTUP MOUNT
ALTER DATABASE OPEN RESETLOGS;

STEP 11: Backup the database.

A BRIEF NOTE ON HADOOP

THE 3 MODES TO RUN HADOOP