Showing posts with label database. Show all posts
Showing posts with label database. Show all posts

Tuesday, August 12, 2014

De-fragmenting indexes in SQL Server

One of the key tasks I have done to improve performance on SQL Server database is defragmenting indexes in a SQL database.

Indexes are very crucial for improving database performance because they make database searches efficient and faster.

Index problems

What are some of the issues indexes may have in a database?
There are some issues usually associated with indexes. Here are some of them:

Indexes with high fragmentation ratio: Indexes with fragmentation will cause a huge performance issue on the database.

Too many indexes on a table: Though indexes are desirable, too much of them may be a problem in a database. Index redundancy can occur from too many indexes on a table. Also, it may cause confusion as to which index to use for sql execution. Also, update statements will take a very long time to complete.

Hypothetical indexes: These are indexes created during a SQL tuning operation to improve performance. They may not be needed after the query is executed.

Unnecessary index: creating an index on a very small table is not recommended because it amounts to misuse of resources. 

Databases identified with one or more tables, with indexes that may require update statistics: If statistics of an index is not updated, there could be a performance issue.

There are foreign keys with no supporting indexes: It is best practice to create an index to support foreign keys.

Indexes have been identified with an index key larger than the recommended size (900 bytes): Index keys should not be greater than 900 bytes.

High number of Full Scans versus Index Searches: This is a clear case of lack of index on a table.


Rebuilding an index

You can use the GUI tool to rebuild an index.



Retrieving Fragmentation info on all indexes

--Use this script to get all the information about all the tables and indexes in a database. You should use this list to prepare a collation of all indexes to defrag in a database. 
You can also use this list to set a threshold of defragmentation based on the trend.

-- Declare database
use Patientdatadb
go

-- Show fragmentation per table
SELECT dbschemas.[name] as 'Schema',
dbtables.[name] as 'Table',
dbindexes.[name] as 'Index',
indexstats.avg_fragmentation_in_percent,
indexstats.page_count
FROM sys.dm_db_index_physical_stats (DB_ID(), NULL, NULL, NULL, NULL) AS indexstats
INNER JOIN sys.tables dbtables on dbtables.[object_id] = indexstats.[object_id]
INNER JOIN sys.schemas dbschemas on dbtables.[schema_id] = dbschemas.[schema_id]
INNER JOIN sys.indexes AS dbindexes ON dbindexes.[object_id] = indexstats.[object_id]
AND indexstats.index_id = dbindexes.index_id
WHERE indexstats.database_id = DB_ID()
ORDER BY indexstats.avg_fragmentation_in_percent desc

Result should list all the indexes on all the tables in each database along with the fragmented percentage. 



DeFragmenting all indexes in a database

To defragment all indexes in a database that are fragmented above a declared threshold, you can use the script below:

Please note that my threshold in my script is set to 30%


-- Declare database
use Patientdatadb
go 
-- Declare variables
SET NOCOUNT ON;
DECLARE @tablename varchar(255);
DECLARE @execstr   varchar(400);
DECLARE @objectid  int;
DECLARE @indexid   int;
DECLARE @frag      decimal;
DECLARE @maxfrag   decimal;

-- Declare maximum fragmentation to allow
SELECT @maxfrag = 30.0;

-- Declare a cursor
DECLARE tables CURSOR FOR
   SELECT TABLE_SCHEMA + '.' + TABLE_NAME
   FROM INFORMATION_SCHEMA.TABLES
   WHERE TABLE_TYPE = 'BASE TABLE';

-- Create the table
CREATE TABLE #fraglist (
   ObjectName char(255),
   ObjectId int,
   IndexName char(255),
   IndexId int,
   Lvl int,
   CountPages int,
   CountRows int,
   MinRecSize int,
   MaxRecSize int,
   AvgRecSize int,
   ForRecCount int,
   Extents int,
   ExtentSwitches int,
   AvgFreeBytes int,
   AvgPageDensity int,
   ScanDensity decimal,
   BestCount int,
   ActualCount int,
   LogicalFrag decimal,
   ExtentFrag decimal);

-- Open the cursor
OPEN tables;

-- Loop through all the tables in the database
FETCH NEXT
   FROM tables
   INTO @tablename;

WHILE @@FETCH_STATUS = 0
BEGIN
-- Do the showcontig of all indexes of the table
   INSERT INTO #fraglist
   EXEC ('DBCC SHOWCONTIG (''' + @tablename + ''')
      WITH FAST, TABLERESULTS, ALL_INDEXES, NO_INFOMSGS');
   FETCH NEXT
      FROM tables
      INTO @tablename;
END;

-- Close and deallocate the cursor
CLOSE tables;
DEALLOCATE tables;

-- Declare the cursor for the list of indexes to be defragged
DECLARE indexes CURSOR FOR
   SELECT ObjectName, ObjectId, IndexId, LogicalFrag
   FROM #fraglist
   WHERE LogicalFrag >= @maxfrag
      AND INDEXPROPERTY (ObjectId, IndexName, 'IndexDepth') > 0;

-- Open the cursor
OPEN indexes;

-- Loop through the indexes
FETCH NEXT
   FROM indexes
   INTO @tablename, @objectid, @indexid, @frag;

WHILE @@FETCH_STATUS = 0
BEGIN
   PRINT 'Executing DBCC INDEXDEFRAG (0, ' + RTRIM(@tablename) + ',
      ' + RTRIM(@indexid) + ') - fragmentation currently '
       + RTRIM(CONVERT(varchar(15),@frag)) + '%';
   SELECT @execstr = 'DBCC INDEXDEFRAG (0, ' + RTRIM(@objectid) + ',
       ' + RTRIM(@indexid) + ')';
   EXEC (@execstr);

   FETCH NEXT
      FROM indexes
      INTO @tablename, @objectid, @indexid, @frag;
END;

-- Close and deallocate the cursor
CLOSE indexes;
DEALLOCATE indexes;

-- Delete the temporary table
DROP TABLE #fraglist;
GO

Friday, August 1, 2014

Bulk data loading into SQL Server Database from Excel

The easiest way to load bulk data into SQL database is the commandline. Open the MS Management Studio New Query and run this command:

BULK INSERT dbo.MyTestDB FROM 'C:\Testfolder\Importfile.txt' 
WITH ( FIELDTERMINATOR =',', FIRSTROW = 2 )

or

BULK INSERT dbo.MyTestDB 
FROM 'C:\Testfolder\Importfile.txt' 
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO

Where dbo.MyTestDB is the table and 'C:\Testfolder\Importfile.txt' is the excel file

Also using MS Management Studio, it is very easy to load bulk data into SQL server database using the Import/Export GUI wizard.

Here are quick steps to follow to load database (csv or xls or xlsx) using GUI:

Step 1: Verify columns
You need to verify the columns on the table and match with data on the excel spreadsheet



Step 2: Run import wizard
Right-click on the table and choose "Import Data". This will launch the "Import/Export" wizard for SQL database.


The import wizard will open and will guide you through the whole data import process. Just follow the guide and you should be fine.



Step 3: Choose Source Host and database
The source host is the server name or hostname of the server where the data will be exported. Once the hostname has been specified, the database should automatically appear among the list in the dropdown. Choose the database you want to export from.


Step 4: Select source Table(s)
At this point, you need to select the source and destination tables. You can choose multiple tables at the same time.
Source is the table where data will be exported/copied from
Destination is the table where data will be imported/loaded


Step 5: Map the columns
At this point, you can map or edit the columns to be imported/exported.
You can use "Edit SQL" to edit how the table will be loaded.
Check "Source" to verify the source table.
Check "Destination" to verify the destination table.
If table is not already existing, SQL can create a new table when you check the radio button "Create destination table".
There is also an option to "Drop and recreate destination table".
To skip column(s), click on the destination column and change to "null".


Step 6: Feedback
This last page shows the confirmation that the data load was successful.

Thanks for reading!

Tuesday, July 29, 2014

Restoring SQL database from a backup file

Restoring a database require skills. You can use commandline or GUI tool to restore a database. Please take note of this:

1. You can restore a database from a backup file (.bak file);
2. You can restore a database from a backup file to a fresh database (database will be created during the process); and
3. You can use database restore from backup as a Data Migration (DM) or Disaster Recovery (DR) plan.

Syntax for restore:
--To Restore an Entire Database from a Full database backup (a Complete Restore):
RESTORE DATABASE { database_name | @database_name_var }
 [ FROM <backup_device> [ ,...n ] ]
 [ WITH
   {
    [ RECOVERY | NORECOVERY | STANDBY =
        {standby_file_name | @standby_file_name_var }
       ]
   | , <general_WITH_options> [ ,...n ]
   | , <replication_WITH_option>
   | , <change_data_capture_WITH_option>
   | , <FILESTREAM_WITH_option>
   | , <service_broker_WITH options>
   | , <point_in_time_WITH_options—RESTORE_DATABASE>
   } [ ,...n ]
 ]
[;]


Example of restore situation:

Using comandline, type the following to restore a database from a backup file:

RESTORE DATABASE TestPatient FROM DISK = 'C:\Testbackup.BAK'
GO

Using the GUI tool is the easiest, but you have to be careful of any option you're choosing.

Step 1: Right click on "Database" menu and select "Restore Database". 
A new dialogue box will appear with various options. This wizard will guide you through the whole database restore process.
NOTE: Make sure your backup file (.bak file) has been copied to the server and you have the required privilege to open the file or write on the file location.


Step 2: This is where you specify the database name. 
If you want to create a new database for this restore process, type in the database name.


Step 3: Source
Under the "Specify source" option, choose "From device" from the radio button and click on the menu on the right corner to expand your options.


Step 4: Add file
Click on "Add" to point the location of your file


Step 5: Choose file
From the box, choose the backup file. The file usually will have a ".bak" extension. Ensure the the correct file is selected, otherwise the restore will fail.
Click "OK" to continue


Step 6: Select file
Once the file the has been selected, click "OK" to move to the next step


Step 7: Restore
At this point, check the "Restore" box to indicate the exact file the database will be restored from, and then click "OK"

Step 8: Run process
Once you click "OK", the restore process runs. You can monitor the restore process through the "Completion" rate on the left tab. It will display in percentage the rate of completion.



Step 9: Done
Once the restore is completely done, you will get a message like this: "The restore of database XXX completed successfully".

Step 10: Verify
To verify that the restore was successfully completed, click on Databases and look from the newly created database in the drop-down.

Thanks for reading!

Friday, June 27, 2014

Managing a database in SQL Server

One of the awesome tools I have used in my many years of managing databases is the Microsoft Management Studio for SQL Server.

LOGIN

To launch SQL Server Management Studio, click on Start > Program > SQL Server Management Studio


Once it launches, you will be prompted for login. Take note of the followings:

Server type: Database Engine
Server name: Name of your server, e.g. BINGO.uacd.edu
Authentication: This could be Windows authentication or Database. You need to create a database user prior to using database authentication. I will address how to create a user in another post.
















CREATING A DATABASE 

After logging into  SQL Server Management Studio, you can create a database by simply right clicking on the database dropdown under the SQL Server Instance. Click on "New Database". A new window pops out and will allow you to name and customize the database.

In this window, you can name the data files and add more options:























TO CREATE A DATABASE USING COMMAND

Login into MS Management Studio, open a New Query Window and run this command:

USE [master]
GO
CREATE DATABASE [kunle] ON  PRIMARY 
( NAME = N'uacd', FILENAME = N'F:\Microsoft SQL Server\MSSQL10_50\MSSQL\DATA\kunle.mdf' , SIZE = 8421632KB , MAXSIZE = UNLIMITED, FILEGROWTH = 102400KB )
 LOG ON 
( NAME = N'uacd_log', FILENAME = N'F:\Microsoft SQL Server\MSSQL10_50\MSSQL\DATA\kunle_1.LDF' , SIZE = 4209664KB , MAXSIZE = 2048GB , FILEGROWTH = 10240KB )
GO
ALTER DATABASE [kunle] SET COMPATIBILITY_LEVEL = 100
GO
IF (1 = FULLTEXTSERVICEPROPERTY('IsFullTextInstalled'))
begin
EXEC [kunle].[dbo].[sp_fulltext_database] @action = 'enable'
end
GO

ADDING OPTIONS

You can also add more options to the newly created database. Login into MS Management Studio, open a New Query Window and run this command:

USE [master]
GO
ALTER DATABASE [kunle] SET ANSI_NULL_DEFAULT OFF 
GO
ALTER DATABASE [kunle] SET ANSI_NULLS OFF 
GO
ALTER DATABASE [kunle] SET ANSI_PADDING OFF 
GO
ALTER DATABASE [kunle] SET ANSI_WARNINGS OFF 
GO
ALTER DATABASE [kunle] SET ARITHABORT OFF 
GO
ALTER DATABASE [kunle] SET AUTO_CLOSE OFF 
GO
ALTER DATABASE [kunle] SET AUTO_CREATE_STATISTICS ON 
GO
ALTER DATABASE [kunle] SET AUTO_SHRINK OFF 
GO
ALTER DATABASE [kunle] SET AUTO_UPDATE_STATISTICS ON 
GO
ALTER DATABASE [kunle] SET CURSOR_CLOSE_ON_COMMIT OFF 
GO
ALTER DATABASE [kunle] SET CURSOR_DEFAULT  GLOBAL 
GO
ALTER DATABASE [kunle] SET CONCAT_NULL_YIELDS_NULL OFF 
GO
ALTER DATABASE [kunle] SET NUMERIC_ROUNDABORT OFF 
GO
ALTER DATABASE [kunle] SET QUOTED_IDENTIFIER OFF 
GO
ALTER DATABASE [kunle] SET RECURSIVE_TRIGGERS OFF 
GO
ALTER DATABASE [kunle] SET  DISABLE_BROKER 
GO
ALTER DATABASE [kunle] SET AUTO_UPDATE_STATISTICS_ASYNC OFF 
GO
ALTER DATABASE [kunle] SET DATE_CORRELATION_OPTIMIZATION OFF 
GO
ALTER DATABASE [kunle] SET TRUSTWORTHY OFF 
GO
ALTER DATABASE [kunle] SET ALLOW_SNAPSHOT_ISOLATION OFF 
GO
ALTER DATABASE [kunle] SET PARAMETERIZATION SIMPLE 
GO
ALTER DATABASE [kunle] SET READ_COMMITTED_SNAPSHOT OFF 
GO
ALTER DATABASE [kunle] SET HONOR_BROKER_PRIORITY OFF 
GO
ALTER DATABASE [kunle] SET  READ_WRITE 
GO
ALTER DATABASE [kunle] SET RECOVERY SIMPLE 
GO
ALTER DATABASE [kunle] SET  MULTI_USER 
GO
ALTER DATABASE [kunle] SET PAGE_VERIFY CHECKSUM  
GO
ALTER DATABASE [kunle] SET DB_CHAINING OFF 
GO


DELETING A DATABASE 

Using command to delete a database, login into MS Management Studio, right click on the database and select "Delete".

This will open another box where you customize the operation as shown the the snapshots.












For command option, login into MS Management Studio and open a New Query Window.









Then issue this command:

USE [master]
GO
EXEC msdb.dbo.sp_delete_database_backuphistory @database_name = N'kunle'
GO
USE [master]
GO
DROP DATABASE [kunle]
GO

Monday, July 8, 2013

Using SQLloader on Oracle 11g

I taught a session on SQLloader over the weekend, so I am going to post the highlights and the exercises I created for the students on this space.

What is Sqlloader?

It is a database tool for loading large volumes of data into the database. It is faster than using an insert and more reliable too. Also, it is free because it comes with the Oracle database utilities.

How to invoke Sqlloader

Lauch sqlloader using the command below:

sqlldr username/password control=controlfile

e.g

sqlldr system/**** control=my_control_file.ctl


Valid Keywords/arguments

    userid --     ORACLE username/password
   control --     control file name
       log --       log file name
       bad --      bad file name
      data --      data file name
   discard --    discard file name
discardmax -- number of discards to allow          (Default all)
      skip --      number of logical records to skip    (Default 0)
      load --      number of logical records to load    (Default all)
    errors --     number of errors to allow            (Default 50)
      rows --     number of rows in conventional path bind array or between direct path data saves
                      (Default: Conventional path 64, Direct path all)
  bindsize --   size of conventional path bind array in bytes  (Default 256000)
    silent --      suppress messages during run (header, errors, discards, partitions)
    direct --      use direct path                      (Default FALSE)
   parfile --      parameter file: name of file that contains parameter specifications



example 1: Use sqlloader to load data into table PERSON

log in as SYSTEM

SQL> create table PERSON
(SSN number(15),
National_ID number(15),
Last_name varchar2(30),
first_name varchar2(15),
sex char(1)
);

create a control file- "person_ctl.txt"

add this.....

LOAD DATA
  INFILE 'c:\person_data.txt'
  BADFILE 'c:\person_data_bad.txt'
  APPEND
  INTO TABLE PERSON
  FIELDS TERMINATED BY ","
  TRAILING NULLCOLS
( SSN,
National_ID,
Last_name,
first_name,
sex 
)

create an infile- "person_data.txt"

add this.....

77770101,7777801,JEFFEREY,KIDMAN,m
77770102,7777802,Clinton R,miller,m
77770103,7777803,ALLEN D,IVES,m
77770104,7777804,Matthew,MAXWELL,f
77770105,7777805,ANTHONY J,REISNER,m
77770106,7777806,CLINT A,KOCH,m
77770107,7777807,ORLANDO,Desantis,m
77770108,7777808,JUAN C,ZIMMERMAN,m
77770109,7777809,STEVE J,EVANS,m
77770110,7777810,Robert D,DAVIS,m
77770111,7777811,Michael,McClendon,m
77770112,7777812,JONATHAN,CONGER,m
77770113,7777813,DAVID F,LOVITT,m
77770114,7777814,Ernesto,FLESHMAN,m
77770115,7777815,Joshua B,pierce,m
77770116,7777816,Justin,OLIVER,m
77770117,7777817,TODD J,Jacobs,m
77770118,7777818,Conner,BROWN,m
77770119,7777819,Dewey,WOOD,m
77770120,7777820,FRANK C,ARAUJO,m
77770121,7777821,SCOTT A,OBYRNE,m
77770122,7777822,JESSE W,HARDIN,m
77770123,7777823,MATTHEW D,CASTOR,m
77770124,7777824,JOSHUA,ANDREWS,m
77770125,7777825,JUSTIN B,Depew,m
77770126,7777826,MATTHEW F,FORST,m
77770127,7777827,ALFONSO L,OGBORN,m
77770128,7777828,COREY J,DINOFF,m
77770129,7777829,CHRISTOPHER J,COMETA,m
77770130,7777830,PEDRO L,WEBB,m
77770131,7777831,ZACHERY R,GUTIERREZ,m
77770132,7777832,George,SAIKU,m
77770133,7777833,WILLIAM P,FISHER,m
77770134,7777834,ANTHONY R,STOWBUNKO,m
77770135,7777835,Terry,GILCHRIST,m
77770136,7777836,ERIC M,LEE,m
77770137,7777837,CHARLES W,MONTGOMERY,m
77770138,7777838,CHRISTOPHER S,SZOKE,m
77770139,7777839,RANDALL J,Konig,m
77770140,7777840,Robert E,TURNBOW,m
77770141,7777841,JUSTIN R,GUTIERREZ,m
77770142,7777842,NICHOLAS P,klein,m
77770143,7777843,ROBERT W,ALLEY,m
77770144,7777844,PAUL D,SANTOCONO,m
77770145,7777845,KORY J,GOMEZ,m
77770146,7777846,WALTER T,STEADMAN,m
77770147,7777847,JOSHUA D,Martin,m
77770148,7777848,CHAN P,GWIZDAK,m
77770149,7777849,CHRISTOPHER W,OKKEN,m
77770150,7777850,NAQUAN D,CARMICHAEL,m


Launch sqlloader from c:\


c:\> sqlldr system/**** control=person_ctl.txt


example 2: Use sqlloader to load data into table SCHOOL

log in as SYSTEM

SQL> create table SCHOOL
(SCH_ID number(15),
SSN number(15),
state_id number(7),
county_id number(7),
county number(7),
state char(2)
);

create a control file- "sch_ctl.txt"

add this.....

LOAD DATA
  INFILE 'c:\sch_data.txt'
  BADFILE 'c:\sch_data_bad.txt'
  APPEND
  INTO TABLE PERSON
  FIELDS TERMINATED BY ","
  TRAILING NULLCOLS
( SCH_ID,
SSN,
state_id,
county_id,
county,
state
)

create an infile- "sch_data.txt"

add this.....

7777801,77770101,262,2020,2020,ga
7777802,77770102,262,2020,2020,ma
7777803,77770103,262,2020,2020,la
7777804,77770104,262,2020,2020,tx
7777805,77770105,262,2020,2020,md
7777806,77770106,262,2020,2020,mi
7777807,77770107,262,2020,2020,mo
7777808,77770108,262,2020,2020,mo
7777809,77770109,262,2020,2020,mi
7777810,77770110,262,2020,2020,tx
7777811,77770111,262,2020,2020,tx
7777812,77770112,262,2020,2020,tx
7777813,77770113,262,2020,2020,md
7777814,77770114,262,2020,2020,md
7777815,77770115,262,2020,2020,sd
7777816,77770116,262,2020,2020,id
7777817,77770117,262,2020,2020,mi
7777818,77770118,262,2020,2020,dc
7777819,77770119,262,2020,2020,dc
7777820,77770120,262,2020,2020,dc
7777821,77770121,262,2020,2020,dc
7777822,77770122,262,2020,2020,dc
7777823,77770123,262,2020,2020,dc
7777824,77770124,262,2020,2020,mi
7777825,77770125,262,2020,2020,dc
7777826,77770126,262,2020,2020,md
7777827,77770127,262,2020,2020,ms
7777828,77770128,262,2020,2020,tn
7777829,77770129,262,2020,2020,tx
7777830,77770130,262,2020,2020,oh
7777831,77770131,262,2020,2020,mi
7777832,77770132,262,2020,2020,az
7777833,77770133,262,2020,2020,ca
7777834,77770134,262,2020,2020,oh
7777835,77770135,262,2020,2020,ga
7777836,77770136,262,2020,2020,ny
7777837,77770137,262,2020,2020,ny
7777838,77770138,262,2020,2020,nj
7777839,77770139,262,2020,2020,nj
7777840,77770140,262,2020,2020,pa


Launch sqlloader from c:\


c:\> sqlldr system/**** control=sch_ctl.txt

NOTE:

To avoid errors, launch SQLldr from the path where your control file is located,
e.g if your control file is c:\database\control1.ctl, then run sqlldr from 'c:\database'

Monday, July 1, 2013

Datapump in Oracle 11g

Datapump import/export are used for data migration. You can export TABLES, SCHEMA, TABLESPACE or DATABASE and restore it on another server.

Quickfacts:

1. Datapump produces a dumpfile with extention .dmp or .imp
2. Privilege 'exp_full_database' is required to export full database
3. Privilege 'imp_full_database' is required to impport full database
4. In datapump, you can user a parfile to store your commands

Getting the help page

expdp help=y

Keyword                                  Description
------------------------------------------------------------------------------
DIRECTORY                          Directory object to be used for dumpfiles and logfiles.
DUMPFILE                             List of destination dump files (expdat.dmp),
                                                 e.g. DUMPFILE=scott1.dmp, scott2.dmp, dmpdir:scott3.dmp.
EXCLUDE                              Exclude specific object types, e.g. EXCLUDE=TABLE:EMP.
FULL                                       Export entire database (N).
HELP                                       Display Help messages (N).
INCLUDE                                Include specific object types, e.g. INCLUDE=TABLE_DATA.
JOB_NAME                            Name of export job to create.
LOGFILE                                 Log file name (export.log).
NETWORK_LINK                  Name of remote database link to the source system.
NOLOGFILE                           Do not write logfile (N).
PARALLEL                             Change the number of active workers for current job.
PARFILE                                 Specify parameter file.
SCHEMAS                              List of schemas to export (login schema).
TABLES                                  Identifies a list of tables to export - one schema only.
TABLESPACES                     Identifies a list of tablespaces to export.
VERSION                              Version of objects to export where valid keywords are:
                                                (COMPATIBLE), LATEST, or any valid database version.
EXPORT

Example 1: Joe is a user on a database, Joe needs to export the whole database.
Solution:
c:\> sqlplus sys/**** as sysdba
     grant exp_full_database to joe;
     commit;
     exit;
c:\> expdp joe/**** full=y dumpfile=joe_database.dmp logfile=joe.log compression=n

^full database is exported
^Dumpfile will be created in the default location, which is 'c:\app\user\admin\orcl\dpdump'


Example 2: System needs to export Joe's schema with compression
Solution:
c:\> expdp system/**** full=n schemas=joe dumpfile=joe_schema.dmp logfile=joe1.log compression=y

^Joe's schema is exported
^Dumpfile will be created in the default location, which is 'c:\app\user\admin\orcl\dpdump'
^Dumpfile will be compressed


Example 3: Joe needs to export 2 tables (Student, Lecturer) to a location on disc ('c:\oracle')
Solution:
c:\> sqlplus joe/****
     create directory oracle as 'c:\oracle';
     commit;
     exit;
c:\> expdp joe/**** full=n tables=Student,Lecturer directory=oracle dumpfile=joe_tables.dmp logfile=joe2.log

^Joe's table are exported
^Dumpfile will be created in 'c:\oracle'
^There should be no space between the tables...e.g tables=(student1,student2,student3)


IMPORT

Example 4: Joe needs to import 2 tables (Student, Lecturer) from a directory called 'oracle'
Solution:
c:\> impdp joe/**** directory=oracle dumpfile=joe_tables.dmp tables=Student,Lecturer logfile=joe3.log

^Joe's table will be imported

Example 5: Joe needs to import 2 tables (Student, Lecturer) and change the tables name to 'student1' and 'student2'
Solution:
c:\> impdp joe/**** directory=oracle dumpfile=joe_tables.dmp logfile=joe3.log remap_table=joe.Student:student1,joe.Lecturer:student2

^Joe's table will be imported and renamed
^Tables must be separated by comma, the colon (:) must separate the oldname and new name


Example 6: System needs to import the whole database from a dumpfile on a flashdrive
Solution:
c:\> cd c:\app\user\admin\orcl\dpdump
Copy dumpfile to the folder
Launch impdp from this location
c:\app\user\admin\orcl\dpdump> impdp system/**** full=y dumpfile=database.dmp logfile=db.log

Example 7: System needs to import joe's schema from dumpfile (of full database) on a flashdrive
Solution:
c:\> cd c:\app\user\admin\orcl\dpdump
Copy dumpfile to the folder
Launch impdp from this location
c:\app\user\admin\orcl\dpdump> impdp system/**** full=n schemas=joe dumpfile=database.dmp logfile=db1.log

^Joe's schema will be imported


Example 8: System needs to import joe's schema and change name to DANIEL and also change the tablespace to daniel's tablespace
from dumpfile (of full database) on a flashdrive
Solution:
c:\> cd c:\app\user\admin\orcl\dpdump
Copy dumpfile to the folder
Launch impdp from this location
c:\app\user\admin\orcl\dpdump> impdp system/**** full=n dumpfile=database.dmp logfile=db2.log remap_schema=joe:daniel remap_tablespace=users:daniel_tbs

^Joe's schema will be imported and changed to Daniel
^Tablespace will be changed from users to Daniel_tbs
^you MUST physically create this tablespace and grant access to Daniel.


Example 9: System needs to import joe's schema using a parameter file
Solution:

create a parfile called 'joe.txt'

USERNAME=system/****
FULL=N
SCHEMA=(joe,daniel,class)
DIRECTORY=oracle
DUMPFILE=database.dmp
LOGFILE=db4.log


Save file and exit

c:\> impdp parfile=joe.txt

Parfile makes it more convenient to adjust datapump commands. It also help when you need to constantly run same commands.

Example 10: System needs to import joe's schema from dumpfile and replace all tables
Solution:
c:\> impdp system/**** full=n schemas=joe dumpfile=database.dmp logfile=db6.log table_exists_action=replace

^Joe's schema will be imported
^Joe's tables will be replaced if they already exist on the database prior to import

Wednesday, June 12, 2013

ORA-24247: network access denied by access control list (ACL) 

Error likes "*Cause: The UTL_HTTP package failed to execute the HTTP request" are very common with Oracle 11g and older versions of Oracle database. The problem is that PL/SQL packages: UTL_TCP, UTL_HTTP, UTL_SMTP, UTL_MAIL, and UTL_INADDR cannot be executed by the current user. 


Applications rely on database schemas to execute functions. For example, using internet maps to determine location in an application requires an external connection to another webpage. This is why the invoker of those packages needs additional privileges to connect to an external host or to resolve the name or the IP address of a host. 

This is how the process works: The packages check the invoker for the necessary privileges only when the calls are made at runtime and raises an exception if the invoker lacks the privileges. 

In Oracle 11.2.0.3.6, this problem was solved. Prior releases of Oracle requires XML DB to be installed and Configured a network Access Control Lists (ACLs) in the database before these packages can work.

What is an ACL? An ACL is a network access control list that enables a DBA control access to external connection through the host. Using an ACL is common in production databases when issues with Network Access like ORA-24247 arises.


Environment
Linux 5.4
Oracle 11.2.0.3.0 (Production)

Cause
Privilege to UTL_http  should have been granted to specific user or schema that will be using the external connection. Though this can be given through patches/updates, it is highly recommended to reduce host vulnerability to manually assign privilege through ACL.
If schema has access to objects after Oracle patch was installed, this is/could be a problem when installed on the production server because accounts that should not have access to this and other objects will have access to things  they are not supposed to.

Solution
Verify that the Port is open
You need to verify that your port is open and can connect externally and also download webpages. Use these commands:

login as user "Oracle" (OS environment)

wget http://maps.google.com/
or
wget -r -l 0 http://maps.google.com/

If you get a response, then your port is open and ready for connection.

Create the user requiring access
Run these commands to create the users needed:

create tablespace patients 
datafile '/home/oracle/app/oracle/oradata/patients/datafile/patients01.dbf' size 50m reuse autoextend on maxsize unlimited;

create smallfile temporary tablespace patientstemp tempfile '/home/oracle/app/oracle/oradata/patients/datafile/patientstemp01.dbf' size 5m autoextend on next 2048k maxsize 1024m extent management local uniform size 1m;

create user patients profile "default" identified by "patients"
default tablespace patients
temporary tablespace patientstemp
quota unlimited on patients account unlock;

grant create session to patients;
grant connect to patients;
grant create trigger to patients;
grant create view to patients;
grant create procedure to patients;
grant create sequence to patients;
grant create table to patients;
grant create synonym to patients;

Run script to create ACL
The script will first drop any existing ACL before creating a new one. This is necessary to reduce or eliminate conflicts.

begin
        DBMS_NETWORK_ACL_ADMIN.DROP_ACL(acl => 'utl_http.xml');
end;
/
commit;
BEGIN
  DBMS_NETWORK_ACL_ADMIN.CREATE_ACL(acl         => 'utl_http.xml',
                                    description => 'geo code ACL',
                                    principal   => 'PATIENTS',
                                    is_grant    => true,
                                    privilege   => 'connect');
  DBMS_NETWORK_ACL_ADMIN.ADD_PRIVILEGE(acl       => 'utl_http.xml',
                                       principal => 'PATIENTS',
                                       is_grant  => true,
                                       privilege => 'resolve');

  DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl  => 'utl_http.xml',
                                    host => 'your_ip_address');
END;
/
COMMIT;

Grant access to user
as user "SYS" run this command

grant execute on utl_http to PATIENTS;

grant execute on DBMS_NETWORK_ACL_ADMIN to PATIENTS;

Assign ACL to networks (in this order)

SQL> BEGIN
  DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl  => 'utl_http.xml', host => 'maps.google.com');
  END;
   /

SQL> BEGIN
 DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl  => 'utl_http.xml', host => 'local.yahooapis.com');
 END;
  /

Verify the ACL

SELECT acl from dba_network_acl_privileges;
/sys/acls/utl_http.xml

SELECT host, lower_port, upper_port, acl FROM   dba_network_acls;

Desc   dba_network_acls;
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------------
 HOST                                      NOT NULL VARCHAR2(1000)
 LOWER_PORT                       NUMBER(5)
 UPPER_PORT                         NUMBER(5)
 ACL                                         VARCHAR2(4000)
 ACLID                                     NOT NULL RAW(16)


Verify permission
Run this script as user "SYS" to verify if the permission to connect externally has been granted to the user - patients:

SELECT acl, principal, privilege, is_grant, 
TO_CHAR(start_date, 'DD-MON-YYYY') AS start_date, TO_CHAR(end_date, 'DD-MON-YYYY') AS end_date FROM   dba_network_acl_privileges;

SQL> SELECT DECODE(
  2  DBMS_NETWORK_ACL_ADMIN.check_privilege('/sys/acls/utl_http.xml', 'PATIENTS', 'connect'),
  3  1, 'GRANTED', 0, 'DENIED', NULL) privilege
  4  from dual;
PRIVILE
-------
GRANTED


Check that it can connect
Run this script as user "SYS" to verify if the permission to connect externally has been granted to the user - patients:

SQL> DECLARE
  2  l_url            VARCHAR2(50) := 'http://maps.google.com';
  3  l_http_request   UTL_HTTP.req;
  4   l_http_response  UTL_HTTP.resp;
  5   BEGIN
  6  l_http_request  := UTL_HTTP.begin_request(l_url);
  7  l_http_response := UTL_HTTP.get_response(l_http_request);
  8  UTL_HTTP.end_response(l_http_response);
  9   END;
10  /

I hope this helps. Please drop your comments or/and questions.

Pix credit: Oracle, NetworkDigest

Tuesday, May 28, 2013

Database auto-start and Stop

Hi folks, I am going to demonstrate in this post the process of writing and launching a script to automatically start and stop a database and listener on Oracle. I am working in this environment:

Oracle 11.2.0.1.0
Linux 5.4
instance name PATIENTS

Purpose: creating an auto-start script on PATIENTS database

Log in as Oracle


vi start_PATIENTS
add this line:

#!/bin/sh
ORACLE_SID=PATIENTS
export ORACLE_SID
PATH=$ORACLE_HOME/bin:$PATH
export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH

echo "Starting Database $ORACLE_SID"
sqlplus /nolog >/dev/null <
     connect / as sysdba
     startup;
     exit
EOF
if [ $? != 0 ] ; then
 echo "Database $ORACLE_SID did not start!"
else
 echo "Database $ORACLE_SID started!"
fi


vi stop_PATIENTS
add this line:

#!/bin/sh
ORACLE_SID=PATIENTS
export ORACLE_SID
PATH=$ORACLE_HOME/bin:$PATH
export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH

echo "Stopping Database $ORACLE_SID"
sqlplus /nolog >/dev/null <
     connect / as sysdba
     shutdown immediate;
     exit
EOF
if [ $? != 0 ] ; then
 echo "Database $ORACLE_SID did not stop!"
else
 echo "Database $ORACLE_SID stopped!"
fi

vi start_listener
add this line:

#!/bin/sh

echo "Starting listener"
lsnrctl start > /dev/null
if [ $? != 0 ] ; then
  echo "listener did not start"
else
  echo "listener started!"
fi


vi stop_listener
add this line:

#!/bin/sh

echo "Stopping listener"
lsnrctl stop > /dev/null
if [ $? != 0 ] ; then
  echo "listener did not stop"
else
  echo "listener stopped!"
fi


change file to executable

[oracle@localhost ~]$ chmod +x start_PATIENTS
[oracle@localhost ~]$ chmod +x stop_PATIENTS
[oracle@localhost ~]$ chmod +x start_listener
[oracle@localhost ~]$ chmod +x stop_listener

launching the script:

[oracle@localhost ~]$ ./start_listener
Starting listener
listener started!

[oracle@localhost ~]$ ./start_PATIENTS
Starting Database PATIENTS
Database PATIENTS started!

[oracle@localhost ~]$ ./stop_PATIENTS
Stopping Database PATIENTS
Database PATIENTS stopped!

[oracle@localhost ~]$ ./stop_listener
Stopping listener
listener stopped!


launching the script:

Log in as Oracle (SSH or Putty)

From /home/oracle, launch the files.

Start database in this order….

Launch start_listener
Launch start_PATIENTS

Stop database this way….

Launch stop_PATIENTS
Launch stop_listener


You should get this message after launching the script…

[oracle@localhost ~]$ ./start_listener
Starting listener
listener started!

[oracle@localhost ~]$ ./start_PATIENTS
Starting Database PATIENTS
Database PATIENTS started!

[oracle@localhost ~]$ ./stop_PATIENTS
Stopping Database PATIENTS
Database PATIENTS stopped!

[oracle@localhost ~]$ ./stop_listener
Stopping listener
listener stopped!

Thanks for reading.