ERP Support Tips

of 61

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of ERP Support Tips

  • 8/8/2019 ERP Support Tips


    Oracle Apps: Art of Supporting Oracle Applications ERP

    Art of Supporting Oracle Applications ERP

    1. Problem Resolution


    The following questions can assist in identifying and resolving problems in a timely


    Has this function worked previously?

    If the answer is yes, then you will need to explore areas such as patch application.

    Has this problem occurred before, during the testing phase of your implementation or

    an upgrade?

    You could review your records at this stage to see if this has occurred previously and

    how it was resolved.

    Has it stopped working as a result of a stand-alone Patch or Patchset being applied?

    At this point it is also a good idea to obtain the current Patchset level you have for

    the particular module.

    Have there been any changes to the configuration of the system technical or


    Technical eg upgrade of the Database, Application, Forms or Reports, Functional eg

    Values in the System Options form have been modified within a particular module.

    Is the problem user specific?

    You will need to identify if other users are experiencing the problem reported or if itrelates to one user only. If it is a single user, things such as personal profile options

    can be reviewed. Or, is it a group of users accessing the same responsibility? Review

    the differences in the responsibilities to try to pin point the issue.

    What environment is it happening in? Is the problem reproducible?

  • 8/8/2019 ERP Support Tips


    Is this a Production, Test, User Acceptance Testing (UAT) or Development system? Is it

    occurring in all, some or only one of the above environments? Try to examine the

    differences between the environments if the problem exists in one but not the other.

    It is also important to try to reproduce the problem in another environment if

    possible. This will aid the problem solving process if the issue needs to be logged withOracle Support Services.

    Is this standard functionality or a customisation of Oracle applications?

    If the problem is standard functionality and the initial research has been done, such

    as referring to MetaLink and User Guides, Technical References and so on, and the

    problem remains unsolved, then a call can be logged with Oracle Support for

    assistance. If it is a customisation made to the application, then it will not be

    supported by Oracle Support Services, and you may need to seek the services of a

    Consulting organisation, if you cannot resolve the problem.

    Reviewing your concurrent log files.

    I cant stress enough the importance of reviewing the log file from your Concurrent

    processes. Even if your concurrent request completed without an error status, it is

    always good practice to just quickly review the log file. Processes such as the Oracle

    Assets Periodic Mass Copy and the General Ledger Journal Import may complete

    successfully, but that does not mean that there were no exceptions encountered.With processes such as these, which transfer information, or with interfaces, always

    review the log file and associated report output for any records that were not

    successful in the process, as they will need to be addressed.

    Always take note of the errors that appear in your log file, and save the file to a

    secure directory, to avoid losing it should the Concurrent Manager requests get



    Before logging a call with Oracle Support Services, the above questions should be

  • 8/8/2019 ERP Support Tips


    answered as well as performing research on MetaLink. MetaLink is a Web support

    service available from Oracle Support Services. It was developed to provide Oracle

    customers with a 24x7 resource to access information from repositories regarding

    support information, including solutions. The following address allows you to register

    on and access MetaLink -

    There are various facets to MetaLink, however the focus of this paper will be on the

    search function for previous reported incidents and the logging, updating and

    reviewing of Technical Assistance Requests (TARS), and the reviewing of Patches, Bugs

    and briefly on the Forums available.

    The Search Function

    The Basic Search

    Enter the search criteria in the search field (To the left of the Search Button). Thenclick on the search button and your results will be displayed.

    The Advanced Search

    The Advanced Search page allows you to refine and further customise your querywhen searching for information on the site. You can narrow your search by selecting

    one or more options. These options include selecting the type of search, the order in

    which information is shown, and the source or area of the site you wish to search.

    To ensure that your search is effective, take some time to review the actual search

    methods available. There are five methods and generally their use will vary on the

  • 8/8/2019 ERP Support Tips


    type of problem/error that you encounter.

    Below is the screen dump of the Advanced search screen.

    This is the information the above search retrieved.

    Match All (AND++) Search

    This is used for general keyword searches and is the default for both basic andadvanced searches.

    Using AND++ without a comma:

    EXAMPLE: Using the keywords,"APP-43446 TAXCODE will return all documents in

    which both "APP-43446" and "TAXCODE" appear. The precise phrase "APP-43446

    TAXCODE" does not have to appear in any of these documents. The more frequently

    the keywords exist in the document the higher the score on the hit list returned.

    Using AND++ with a comma:All words BEFORE the COMMA must exist, while each word AFTER the comma scores

    higher if present.

    EXAMPLE: Using the keywords, "APP-43446 unable to find, taxcode for this

    transaction" will return all documents in which "APP-43466 unable to find" appear but

    "taxcode for this transaction" are optional. All documents with the words "APP-43446

    unable to find" will appear in the hit list but those documents with "taxcode for this

  • 8/8/2019 ERP Support Tips


    transaction" also appearing in the document will score higher than those without

    these optional words.

    ConText Syntax Search

    Use the ConText Syntax option if you want to conduct a detailed search that involves

    more than keywords. This option searches the site using the ConText Query language.The ConText Query language uses special search operators that define terms within a

    query expression.

    The following table provides examples for some commonly used search operators:


    Input Result Description

    % Wildcard Character 10565%

    te%st 10565.6

    test, teeniest, tenderest The percent symbol matches any substring.

    ( _ ) Wildcard Character 80_86

    _est 80186,80286, 80z86

    best, jest, pest, rest, test The underscore matches any single character.

    ( ) Group Character (peter, or paul and mary) peter and/or paul and mary The open

    parenthesis indicates the beginning of a group. The first close parenthesis

    encountered is the end of the group. Any open parentheses encountered before the

    close parenthesis indicate nested groups.

    { } Group Character {end-of-file}{ORA-600} end of file

    ORA-600 Performs the same function as regular brackets except it prevents the

    penetration of other expansion operators.

    $ Stem Expansion $scream scream screaming screamed Expands a query to include all

    terms having the same stem or root word as the specified term.

    ? Fuzzy Expansion ?feline feline defined filtering Expands a query to include all terms

    with similar spellings as the specified term (English-language text only).

    Match any (OR) Search

    Use the Match any (OR) option to find information on multiple topics or for synonyms.

    The Match any (OR) option is a Boolean expression that allows you to find documents

    in which either of two or more search words appear, again without needing to know

    the ConText search search syntax.

  • 8/8/2019 ERP Support Tips


    EXAMPLE: Using the keywords, APP-43446 Taxcode will find all documents in which

    APP-43446 appears and all documents in which Taxcode appears.

    Doc ID Search

    Querying by document ID number allows you to retrieve a specific document fromOracle Support Services' databases. A document ID number is a unique identification

    number assigned to every file stored in Oracle's Support Knowledge Base. Document ID

    numbers can be obtained from an Oracle analyst or by looking at the document ID in

    hitlists from other types of MetaLink searches.

    Weighted (FUZZY) Search

    Use the Weighted (FUZZY) search option to find documents that contain any of the

    keywords (like a Match any (OR) search) but by using a comma you can rank the

    importance of the keywords.

    EXAMPLE: Using the keywords, "ORA-942, AR_STATEMENT_HISTORY" would find

    documents that contain either of those keywords, however it would rank "ORA-942" as

    more important than AR_STATEMENT_HISTORY. Scores take into account word

    frequency so it is possible that an article containing 10 occurrences of

    AR_STATEMENT_HISTORY would score more than an article containing both terms.

    Technical Assistance Request (TAR) - Create, Update and Modify

    With the correct privileges, you have the ability to create, update and modify the

    TARS in Metalink. When you create an internet TAR (iTAR), it is sent to a queue in the

    Metalink System. All TARS are transferred into the internal tracking system every 15

    minutes and an analyst will then be assigned to work on it.

    Below is a screen dump of a Global TAR search by a single Customer Support Identifier

    (CSI) Or Multiple CSIs within the same country.

  • 8/8/2019 ERP Support Tips


    Below is a screen dump representing a search for a specific TAR number.


    A patch is a collection of fixed bugs assembled and tested by support analysts and

    developers within Oracle. If you find a problem that has been reported before and

    there is a patch available you can download the patch directly from the web site. You

    can also view a list of available patches for a product and platform.

    Type in the desired patch that you wish to download, the release, product, platform,language and type. If you know the file that the problem occurred in, then type this

    in the Includes File box. Ie Statement Generation program file arxsgp.lpc. Then it will

    list all the patches that contain this file. You can choose to order it by Patch Number

    or Release Date. Then click submit. It is very important to read the README.txt as

    there could be pre-requisite or other important instructions you need to follow.


    These are issues that are logged into the bug database and this is the primary tool

    used by Development to determine the problem and devise a solution. The Bug

    Database Search page allows you to specify your search either by a bug number or by

    other search criteria such as product version, platform, or status. You can also

  • 8/8/2019 ERP Support Tips


    determine the order of your query results by sorting how you would like the

    information displayed.

    If you can not retrieve a bug then it may be due to the following reasons;

    1) The bug may not be published

    2) Bugs have to be newer than June 1998

    3) You can only view bugs that match the products and platforms that are licensed

    according to the CSIs you have inserted in the Show Licence section of Metalink.

    This will retrieve the history of the bug and provide a resolution.


    A forum is an interactive area for discussions and commentaries dedicated to a

    certain topic. Forums allow you to post questions and comments and receive

    responses within 2 business days from a Support Analyst who has expertise in that

    product area. If this is not obtainable within 2 days then advice from Oracle will be

    given. It also gives you the opportunity to reply to questions posted by other people.

    The forums are intended to address problems with a severity level 2,3 or 4. If the

    problem is a severity 1 then you must make a call to support. Forums are threaded, so

    a reply to a particular posting becomes part of the thread.

    Select the relevant Forum Product Groups from the drop-down list.

    Click on the type of Thread: All, New, Changed.

    All Threads posted since the date indicated in the posting period.

    New- Only those threads posted since your last visit.Changed- Only those threads with changes ie new replies

    From the time drop down list, select the number of days for which you would like to

    view messages.

    Option to click on Only Threads in which I participate, that is the forum threads

    which you have posted a message or replied to a message.

  • 8/8/2019 ERP Support Tips


    Click on the Open Forum Button

    For Further information about Metalink please review- Metalink Quick Reference

    Sheet. Document id 115465.1


    The Examine utility can be very useful to view values of hidden fields in a form that

    relate directly to columns in the Oracle tables in your database. It is often used to

    find the value of an Id column, which is usually the primary key used to identify

    records in a table.

    The Navigation path for this is as follows -

    Release 10.7 Character

    \Other Debug

    Release 10.7 GUI 11

    Help > Tools > Examine

  • 8/8/2019 ERP Support Tips


    Release 11iHelp > Diagnostics > Examine


    This profile determines if the diagnostic utility Examine requires an Oracle database

    password. It also controls access to most of the other functions on the Tools menu.

    1) The setting of Yes means that you can automatically use the Examine tool when

    the choice is on the menu.

    2) The setting of No mans that you must enter a password to use the Examine tool

    when it appears on the menu.

    Your System Administrator controls the setting of this profile option.

    You may have noticed on your Tools menu in Rel 10.7SC, that there is also a debug

    option available. This will effectively run your form in debug mode, displaying debug

    messages as each step executes. Please note that this option does not work with

    version 4.5 of Forms, but is fixed in version 6.


    It is important to identify the current version of the program that is experiencing the

    problem, as a bug in the version of the program you are running may be the cause of

    your problem.


    A Form version is easily obtained via the front-end application. The navigation path is

    as follows

    Release 10.7 Character

  • 8/8/2019 ERP Support Tips


    \Help Version

    Release 10.7 GUI 11i

    Help > About Oracle Applications

    This will provide you with several pieces of useful information, including the Database

    version, the Applications version, the Form Name and Form Version.


    To obtain the version of a report or concurrent executable, you need to firstly log on

    to the operating system, and go to the appropriate directory -

    For Reports:

    Release 10.7

    Cd $/srw

    Release 11

    Cd $/reports

    For a Spawned Concurrent program:

    Cd $/bin

    Issue the following command to retrieve the version number of the report program or

    the libraries used by a concurrent executable


    Strings -a |grep Header


    Ident |more

  • 8/8/2019 ERP Support Tips



    Find /i Header


    If the problem is occurring in a standard Oracle Form, obtain the following

    information by navigating to Help-> About Oracle Applications in the Forms menu -

    Oracle Forms Version (example, this will help determine the form

    patchset level.

    Form Name: (example FNDSCSGN), will assist in finding previous form issues.

    Individual Form Version (example 11.0.30), this is important if reporting a bug to

    development is necessary.

    You can try to regenerate the Oracle Standard Forms (All platforms) and relink form

    executables (UNIX Only)

    Regenerate the forms that are causing the problems. Forms can be regenerated by

    either using the adadmin installation utility, or manually at the command line. If you

    are on a UNIX platform, you will also need to relink the form executables.

    Refer to the Installations Manuals that are appropriate for your Release and Platform.


    Sometimes there is insufficient information in the Concurrent log file to determine

    the reason why a particular concurrent process is failing or completing with errors. Inthis situation, if a TAR is logged with Oracle Support Services, they will quite often

    request you to run the process from the command line. This will also rule out if there

    is in fact a concurrent manager problem. To do this you need to firstly take note of

    the exact parameters as listed in the parameters field when you view your concurrent

    request. Then, go to the command line in your operating system and change directory

    to where the program resides: Cd $/sub-directory.

  • 8/8/2019 ERP Support Tips


    Executable Program

    Enter the following command, substituting the parameters from the Concurrent

    Manager, and the userid and password:

    0 Y Y

    is the parameters you see from that concurrent request

    Note: when you run a concurrent program executable from the command line, you

    should use a space instead of a comma to separate between each parameter.

    / is the username/password of the apps account for your database.

    Eg For the General Ledger Posting program, if the parameters you see in the

    Concurrent request are 1, 101, 2546, you would enter the following command

    GLPPOS 0 Y 1 101 2546 Y

    This will execute the process and produce a debug log file in the format Lxxxxx.log.

    This log file will reside under the directory. The information in this log file can give

    Oracle Support Services a better idea of what may be causing the problem.


    Report programs reside in:

    Product_TOP/srw (Apps 10.7)

    Product_TOP/reports (Apps 11)

    To run a report from the command line, issue the following command -


    r25run userid= report= .rdf

    destype= desname=desformat=batch=yes

  • 8/8/2019 ERP Support Tips



    In NT, use the r25run32.exe executable to run a report:

    r25run32 userid=apps/[email protected] report=.rdf

    destype= desname=batch=yes



    SQL*Trace is a facility which assists you in identifying the SQL statements that are

    executed by a process. Additionally, it provides you with statistical information for

    further analysis, which can be useful when there are performance issues or problems

    with database objects.

    There are four levels in your Oracle Applications at which SQL*Trace can be enabled




    Profile (applicable to Release 10.7 only)

    Database Trace

    A Database level trace requires a shutdown and startup of both the Concurrent

    Managers and the Database. If there are a number of users on the system or the

    system is unavailable due to 24 x 7 requirements, a database level trace may not

    always be feasible.

    The Database trace process is as follows

    1) Perform a shut down of the concurrent managers, then the database.

    2) Modify the start up parameter file (init.ora) to ensure the following initialisation

    parameters are set


    USER_DUMP_DEST (destination directory for trace output)

  • 8/8/2019 ERP Support Tips



    3) Restart the database, then the concurrent managers.

    4) Run the process for which the problem is occurring.

    5) Retrieve the trace file(s) from the USER_DUMP_DEST directory.6) Perform a shut down of the concurrent managers, then the database.

    7) Restore the original init.ora file.

    8) Restart the database, then the concurrent managers.

    Steps 1-3 and 6-8 above should always be performed by your Database Administrator.

    The value of the initialisation parameter SQL_TRACE dictates whether the SQL trace

    facility is enabled or disabled when you begin a session in ORACLE. Its presence in the

    init.ora file sets the initial value for this trace. This will apply to all sessions on an

    ORACLE instance. You can additionally enable or disable the SQL trace facility just for

    a specific session with the SQL_TRACE option of the ALTER SESSION command. This

    allows you to obtain smaller and more readable trace files and is particularly useful

    for developers. The SQL statement to achieve this is



    Form Trace

    A Form level trace requires the Trace option to be available as a menu item. The

    trace process is as follows

    - Navigate to the Form for which you wish to set the Trace.- Turn the Trace feature on when you are at the point in the form that you want to

    turn trace on

    GUI versions

    Help > Tools > Trace

  • 8/8/2019 ERP Support Tips


    Rel 10.7 Character

    \ Other Trace On

    Rel 11i

    Help > Diagnostics > Trace

    - Note the time trace is started.

    - Execute the actions you wish to trace.

    - Turn the Trace feature off. Make sure this step is done, or you will severely degrade

    your system performance.

    GUI versions

    Help > Tools > Trace

    Rel 10.7 Character

    \Other Trace Off

    Rel 11i

    Help > Diagnostics > Trace

    - Locate the trace file(s) with the time stamp when you ran your trace. The trace file

    will be in the USER_DUMP_DEST directory.

    Note: if a trace file is being generated to track performance, the parameter

    TIMED_STATISTICS in the init.ora file must be set to TRUE.

    Report TraceA Report level trace would generally require a developer or technical resource, and

    involves the following steps

    1) Logon as applmgr and change directory to where your .rdf file resides.

    2) Create a backup copy of your current .rdf and .rex files in the appropriate srw

    directory. Create the .rex text of the report if necessary.

  • 8/8/2019 ERP Support Tips


    3) If the .rex does not exist, you can easily regenerate the report executable by doing

    the following:


    R25convm userid=/ source=stype=rdffile dtype=rexfile dest=

    overwrite=yes batch=yes


    R25conv32 syntax:

    R25convm userid=/ source=

    stype=rdffile dtype=rexfile dest=

    overwrite=yes batch=yes

    4) Modify the .rex file for the report.

    Include an additional line after the statement FND SRWINIT in the Before Report


    srw.do_sql (alter session set sql_trace=TRUE)

    This will activate the SQL Trace in the report

    Include an additional line after the statement FND SRWEXIT in the After Report


    srw.do_sql (alter session set sql_trace=FALSE)

    This will deactivate the SQL Trace in the report after the report completes.

    5) Convert the modified .rex file into an .rdf file


    R25convm userid=/ source=

    stype=rexfile dtype=rdffile dest=

    overwrite=yes batch=yes

  • 8/8/2019 ERP Support Tips



    R25conv32 syntax:

    R25convm userid=/ source=

    stype=rexfile dtype=rdffile dest=overwrite=yes batch=yes

    6) Run the report.

    7) Locate the trace file(s) from the USER_DUMP_DEST directory.

    There is another method that can be used to trace an individual report within Oracle

    Applications for Release 10.7:

    In Character Mode:

    1) Log on as System Administrator.

    2) \Navigate Concurrent Program Define.

    3) Enter Query mode.

    4) Enter your report-name in the short name field.

    5) Execute Query.

    6) Navigate to the next block.

    7) Cursor is now in the concurrent program details.

    8) Select Parameters.9) Arrow down through the parameters, looking for the name = 'trace'.

    If not found, then create a new parameter:

    Add sequence number, such as 999

    Name = Trace

    Description = Report Trace

    Enabled = YES

    Value Set = Yes_No

    Required = NoEnable Security = No

    Default Type = Profile

    Default Value= NO

    10) Save the record

    11) Run the report, making sure that profile option trace = YES.

  • 8/8/2019 ERP Support Tips


    In GUI Mode:

    1) Log on as System Administrator.

    2) Navigate to Concurrent>Program>Define.

    3) Enter Query mode.

    4) Enter your report_name in the short name field5) Execute Query.

    6) Click on the [Parameters] button.

    7) Search through parameters, looking for parameter name = trace.

    If not found, then create a new parameter:

    Enter Seq = 999

    Parameter = trace

    Description = report trace

    Click on Enabled

    Value Set = Y or N

    Default Type = profile

    Default Value= No

    Required = no

    Enable Security = no

    8) Save the record

    9) Run the report, making sure that profile option trace = Y.

    Utilities:SQL TraceSQL trace files can also be generated for concurrent programs through the use of this

    profile option. The trace can be enabled at all levels, i.e. Site, Application,

    Responsibility and User, by setting the profile Utilities:SQL Trace to Yes. This

    option is controlled by your System Administrator - users can see the profile option

    but cannot update it.

    This profile option is only applicable to Release 10.7.

    In Release 11, the equivalent trace utility for concurrent programs is set within each

    individual Concurrent Program definition. It is only the System Administrator or

    Application Developer responsibility that has access to define concurrent programs.

    The information produced by a SQL*Trace is stored in output files with a .trc

  • 8/8/2019 ERP Support Tips


    extension. As the format of the trace output is difficult to read, there is a utility

    available called TKPROF which should be applied to the trace file(s) to reformat the

    output in a more readable format.

    The basic command syntax of the TKPROF utility is as follows

    $ tkprof

    If you do not know the destination directory for the creation of your trace files, this

    value can be found easily through the following SQL query using SQL*Plus

    SELECT value

    FROM v$parameter

    WHERE name = 'user_dump_dest';


    There are several profile options across the Oracle Applications that can assist with

    problem resolution. These profile options serve to provide more information in the

    Concurrent log file in the form of debug tracing messages

    Product Updateable Profile Option CommentsOracle Cost Management At all levels for the System Administrator

    CST:Cost update debug level

    (Release 10.7 11i) For the cost update program, available values are as follows

    None Do not print debug messages.

    Regular Print a debug message for each subroutine executed.Extended Print a debug message for each SQL statement executed.

    Full Print a debug message for each SQL statement executed and keep any temporary

    data in the database

    Oracle Master Scheduling/MRP At all levels MRP:Trace Mode

    (Release 10.7 11i) Indicates whether to enable the trace option. Available values are

  • 8/8/2019 ERP Support Tips


    Yes or No

    At all levels MRP:Debug Mode

    (Release 10.7 11i) Indicates whether to enable debug messages within MRP and

    Oracle Work in Process. Available values are Yes or NoOracle Assets Application


    User FA: Print Debug

    (Release 10.7 - 11i) This profile option is used by Oracle Support Services as a tool to

    identify a problem with the code. Available values are Yes or No



    User FA: Print Timing Diagnostics

    (Release 10.7 - 11i) This profile option is used by Oracle Support Services as a tool to

    identify a problem with the code. Available values are Yes or No

    Oracle Order Entry Application


    User OE:Debug

    (Release 10.7 11i) Indicates whether debug messages are displayed on forms.Available values are Yes or No



    User OE:Debug Trace

    (Release 10.7 11i) This profile option is used by Oracle Support Services as a tool to

    identify a problem with the code. Indicates whether a trace file is created during a

    Concurrent Program execution. Available values are Yes or NoOracle Work in Process At all levels TP:WIP Concurrent Message Level

    (Release 10.7 - 11i) Determines the level of detail reported in the move transaction

    concurrent log file during move and resource transaction validation/processing.

    Available values are

  • 8/8/2019 ERP Support Tips


    Message level 0 Reports only errors.

    Message level 1 Reports processing activities and errors.

    Message level 2 Reports and time stamps processing activities and errors.

    Oracle General Ledger At all levels FSG: Message Detail

    (Release 10.7 11i) Determines the level of detail in your error message log file for

    FSG reports. Error messages are divided into three categories

    Catalog I contains detailed statistics useful for program debugging

    Catalog II contains process messages useful for finding out where a process failed

    Catalog III contains only error messages

    Available values for this profile option are

    None No messages.

    Minimal Catalog III messages.

    Normal Catalog II and III messages.

    Full Catalog I, II and III messages.

    Must be set by System Administrator. GL:Debug Directory

    (Release 11-11i) Specifies the directory where the Workflow debug files are created.The database must have write permissions to the specified directory. Available values


    Yes The database may create debug files in the nominated directory.

    No The database cannot create debug files.

    Oracle Projects Application


    User PA:Debug Mode

    (Release 11 -11i) Determines if PA processes and reports are run in debug mode.

    Available values are Yes or No.

    Oracle Purchasing At all levels PO:Set Debug Concurrent ON

    (Release 11 11i) This profile option is used for finding problems with Requisition

  • 8/8/2019 ERP Support Tips


    Import. For performance reasons, the value should ALWAYS be set to No, unless

    otherwise instructed by Oracle Support Services for debugging purposes.

    At all levels PO:Set Debug Workflow ON

    (Release 11 11i) This profile option is used for finding problems with PurchasingWorkflow processes. For performance reasons, the value should ALWAYS be set to No,

    unless otherwise instructed by Oracle Support Services for debugging purposes.

    Account Generator At all levels Account Generator:Run in Debug Mode

    (Release 11 11i) Determines if Account Generator processes are run in debug mode.

    Available values are Yes or No.


    Signal errors are how Unix processes communicate with each other and the operating

    system. This is not a problem with the operating system, but is part of the operating

    systems mechanism of protecting itself. When programs terminate abnormally, this is

    communicated to other processes and the kernel by means of a Signal code. The

    actual code associated with the signal indicates the nature of the termination. The

    meaning of the signal message is contained in a file called signal.h, which is located

    usually in the /usr/include/sys sub-directory.

    Some of the more common Signal errors are as follows

    SIGNAL 4

    A Signal 4 error usually indicates an illegal/unknown instruction was encountered.

    This is typically caused by bugs in the code. Explore Metalink for the error to find out

    if your particular problem is related to a known bug for which there may be a patch.

    Some encounters with a Signal 4 error, for which there are patches available, include

    Process Possible Cause Possible Resolution

    AR to GL Transfer In Rel 10.7 Product 16.1, you may only have 5 accounting periods

    open at one time. Fixed in Patch 932703 or in 11.02 Patch 713261. Fix is also included

    in 11.0.AR.D for Rel 11.

  • 8/8/2019 ERP Support Tips


    FA Mass Revaluation Preview Report Possible cause is not having set up the

    Revaluation Reserve and Amortization Accounts for all Categories/Books. Perform

    relevant FA setup

    Open and Close GL periods Retained Earnings accounts contain disabled or expired

    segments Apply database Server side patch 642941 for Rel 10.7. This will providemore meaningful error messages on the setup issues/steps not completed prior to

    running the Open Next Period process.

    SIGNAL 10

    A Signal 10 usually suggests a memory or addressing error, but it can also be caused

    by a bug in the code, or if an .rdf file has been incorrectly ftpd from the client in

    ASCII mode. There are several patches available on Metalink for Signal 10 errors

    caused by bugs in the various Oracle products. If it is a memory-related problem,

    sometimes shutting down the Concurrent Managers and the database, and booting the

    machine will eliminate any run away processes and reclaim resources.

    Examples of Signal 10 errors include -

    Process Possible Cause Possible Resolution

    AP Mass Additions Create Lack of space in the database. Ask your DBA to check the

    available space in the FA data and index tablespaces, and allocated an additional

    datafile if necessary.Oracle Alert process If email (Unix Sendmail) is part of the action items defined in the

    Alert definition, and Sendmail is not configured correctly for the system, is non-

    functional or is missing. Ask your Unix Administrator to verify the Sendmail

    configuration, as it is specific to certain variations of Unix.


    A Signal 11 error indicates that a memory fault has occurred, where an Oracle

    executable has attempted to access memory incorrectly, e.g. program

    pointers/addresses for memory allocations are incorrect or corrupted, or an attempthas been made to use a segment of memory for which it does not have permission.

    The most common causes of a Signal 11 are

    1) Data Corruption

    2) File Size insufficient disk space in $APPLCSF/out directory where concurrent

    manager output is created

  • 8/8/2019 ERP Support Tips


    3) Kernel parameter problems

    4) Lack of storage space in Tablespaces, Rollback Segments etc

    5) Reports ftpd using incorrect mode ASCII instead of BINARY

    Examples of Signal 11 errors include -

    Process Possible Cause Possible Resolution

    FA Periodic Mass Copy Header file corruption Fixed with Patches 924617 and 958462.

    FA Mass Change Bug in Mass Change program

    For Rel 10.7 -Fixed with Patch 534568 (or Patchset C or higher)

    For Rel 11 Fixed with 709527 (or Patchset 11.0.FA.C part of the 11.0.3

    Maintenance Pack)

    The problem with Signal errors is that whilst we know the meaning of the Signal code

    that was encountered, how do we pinpoint which of the possible causes actually

    caused this violation ?

    What to look for with Signal errors

    Trap the Core Dump file !!

    When a Signal error is encountered by the system, a core dump is usually produced.

    This core dump is created at the instant the process was aborted, and is produced in

    the form of a file called core. The information contained in the core dump file is

    most valuable in helping to resolve a Signal error. It is important to trap the file as

    soon as it is created, as these types of files are overwritten the next time a core

    dump occurs. It is always a good idea to rename the core dump file produced and

    move it to a different directory.

    The core dump file is usually contained in the directory where the command was

    issued, or in $ORACLE_HOME/dbs.

    If a core dump is not generated, this is usually due to the dump destination directory

    being full, or the process not having write access to the dump directory.

  • 8/8/2019 ERP Support Tips


    Produce a debugged version of the Core Dump file !!

    You can relink the executable in debug mode by using the Adrelink utility. This will

    allow you to rerun the process causing the Signal error, producing a debugged version

    of the core dump. This provides more comprehensive statistics than the standard core

    dump file produced without debug switched on.

    An example of using the debug feature is outlined below, for the Oracle Assets

    Periodic Mass Copy process (executable FAMCP)

    1) Log in as APPLMGR to the operating system and environment affected.

    2) Change directory to the directory in which the executable resides e.g.


    3) Make a backup copy of the current executable.

    4) Issue the following command -

    Adrelink force=y ranlib=y link_debug=y fa FAMCP

    This will produce a log file called adrelink.log, which can be located in

    $APPL_TOP/install/log. This log file will be appended to every time a relink command

    is initiated.

    5) Re-run the process that produced the Signal error, e.g. Periodic Mass Copy.

    6) Turn debug off by typing in the following command

    Adrelink force=y ranlib=y link_debug=n fa FAMCP

    The problem with a core dump file is that it is not easily decipherable. You will need

    to enlist the help of your DBA or a technical resource with expertise in this area, as

    you will need to determine which debugger exists on your system, and use it to

    produce a stack trace from the core dump. The Unix command script should be

    used to capture the output of the debugger. This can then be sent to Oracle Support

    Services to help them determine why the core dump occurred.

    Checklist for Signal errors

    The following general checklist, utilising the techniques described earlier in this

    White paper, can be used as a guide to trying to determine the cause of a Signal error

  • 8/8/2019 ERP Support Tips



    1. Review Concurrent Request log for any useful information on the error encountered

    2. Locate core dump file rename and move to a separate directory

    3. Identify full version numbers of the product, RDBMS and the program thatencountered the error

    4. Answer the basic questions

    Has the process worked before ? If so, when, and has anything changed since then,

    such as application of patches, upgrades or configuration ?

    What is the current Patch Set level for the product?

    Is the error reproducible in the same environment ?

    Is the error reproducible in another environment ?

    Is the problem related to a specific User or Responsibility?

    If using Release 10.7SC, is the error reproducible in Character mode ?

    Is it standard or customised functionality ? If its a custom report, and is based on a

    standard report, run the standard report to see if it too errors

    5. Check Metalink for known bugs relating to this Signal code and any patches


    6. Check storage space in tablespaces, rollback segments, etc

    7. Bounce the Concurrent Manager

    8. Run the process from the operating system command line

    9. If Signal error occurred when running a report :1) Check disk space in $APPLCSF/out directory

    2) Check report was ftpd in BINARY mode

    3) Check SRW.USER_EXIT (FND SRWINIT) in Before Report Trigger

    4) Check SRW.USER_EXIT (FND SRWEXIT) in After Report Trigger

    5) Check for P_CONC_REQUEST_ID parameter in user parameters

    10. Rerun the process in debug mode

    1) Application debug mode e.g. FA:Print Debug (profile option)

    2) ADRELINK utility11. Check for data corruption:

    1) Check parameters entered when submitting the Concurrent Request. If it is a

    report process, try narrowing down the parameters e.g. smaller date range

    2) If submitted by batch feed, check input file for errors in field mapping or the

    presence of invalid characters

    3) Process of elimination. Identify the records being processed. Run the SQL

  • 8/8/2019 ERP Support Tips


    statements being executed by the program, and check the status of each record for

    data corruption.

    12. Log TAR with Oracle Support Services

    13. Send debugged core dump file and concurrent log file to1) Oracle Support Services

    2) DBA

    Include all relevant version numbers


    Asset Trace

    This SQL script is designed to provide detailed information on a single asset. It is

    extremely useful for troubleshooting problems with your assets. Whenever you log a

    TAR with Oracle Support Services for problems with particular assets, you will ALWAYS

    be asked to run this script for the offending asset(s). The script captures the data

    contained in all the Oracle Assets tables for an individual asset, and outputs the

    details to a spool file. You should find this file in the admin/sql directory under

    $FA_TOP. To run the script, log into SQL*Plus as the APPS user and type -

    @trace.sql or start trace.sql

    You will be prompted to enter the ASSET_ID and BOOK_TYPE_CODE [uppercase].

    To get the ASSET_ID run the following statement in SQL*Plus -

    select asset_id from fa_additions where asset_number = '[the asset number]';

    If you cannot find the script on your system, you can locate it on Metalink, or

    alternatively contact Oracle Support Services and get them to e-mail the script to

    you. The script is called trace10.sql (Rel 10) or trace11.sql (Rel 11), and produces anoutput file called t107.lis and t11.lis respectively. There is also a tracehtml.sql that

    produces the output in html format (fa_trc.html), which can be viewed using IE

    Explorer 4.0+ or Netscape 4.0+. This version of the trace script works with both

    Release 10 and Release 11 of Oracle Assets.

    To effectively analyse the asset data in the trace output, you really need to have a

  • 8/8/2019 ERP Support Tips


    good understanding of the Oracle Asset tables in order to determine if there is data

    corruption. Alternatively, log a TAR with Oracle Support Services and send the trace

    results to them for further investigation.

    Profile Option FA:Deprn SingleThis profile option should be used specifically when a depreciation run fails in Oracle

    Assets. It controls the caching buffer used when depreciation is run. You can set the

    buffer to either No or Yes. If you set the value to No, then the cache is reset

    after every 20 assets. If the value is set to Yes, then the cache is reset after EVERY


    When depreciation encounters an error in Rel 10.7 and 11, the program will fail at

    that point and rollback any depreciation that was committed. In this situation, change

    the value of this option to Yes under the Fixed Assets Manager responsibility, and

    the value of the FA: Print Debug profile option to Yes, and rerun depreciation - the

    process will fail again, but you will find that the log file now produces detailed

    information on the actual asset causing the failure. Once your resolve this, you will be

    able to resubmit the depreciation process. If there are no other errors, the process

    will complete successfully.

    In Release 11i, the run depreciation and close periods have been broken out to

    separate processes. If depreciation fails, and you set the FA:Deprn Single value toYes, it will process all the assets in your Depreciation Book and capture ALL errored

    assets in the log file. This way you can fix all of your errors in one pass, resubmit

    depreciation, and this time only the corrected assets will process.

    Please note you should only ever change the value of this profile option to Yes when

    depreciation fails.

    ap375496.sql AP Trial Balance Rebuild scriptThis script can be useful if your AP Trial Balance report is not reporting correctly on

    outstanding liability. The script is a data fix script that can be run to recreate the

    trial balance records in the AP_TRIAL_BALANCE table for all records (payments and

    invoice distributions) that have been transferred to GL within a specified range of

    accounting dates. This script is contained in patch 375496 and can be downloaded

    from Metalink.

  • 8/8/2019 ERP Support Tips


    Note the script is not needed for cash basis installations.

    Account Generator FA Test Script

    This script assists in diagnosing problems in Oracle Assets concurrent processes thatfail due to account generation. It replaces the Flexbuilder Test screen used in Release

    10. The output from the script tells the User the following

    1) The code combination id that Account Generator attempted to build.

    2) The reason why Account Generator could not build this code combination.

    3) If Dynamic insertion is set to Yes.

    When you run the script, it will prompt you for a number of parameters. All the

    information that feeds this script is contained in the log file from the failed run.

    Release 11 of Oracle Assets is delivered with this SQL*Plus test script called

    faxagtst.sql. You should find this in the $FA_TOP/admin/sql directory. The latest

    script comes with 11.03 Minipack. Contact Oracle Support Services for the latest


    Account Generator Purchasing Workflow Test Script

    In order to find out what is wrong with the Account Generator workflow, a script

    called wfstatus.sql can be run. The parameter WF_ITEM_KEY is required to run thisscript. The wfstatus.sql gives important debug messages, including error messages for

    the processes that are erroring. This provides the same information that the Workflow

    Monitor does, if the customer has Workflow Monitor set up. Running the script

    involves the following steps

    1) Check the profile option Account Generator: Run in Debug mode. This should be set

    to Yes. (No or Null value means that the account generator process is being run in

    'Synch' mode, in which case no information is logged in the workflow tables, andhence wfstatus.sql would not retrieve any information)

    2) After setting the profile option to Yes, simulate the error, by navigating to the

    Distribution zone of Enter Purchase Order or Enter Requisition, and trying to build the

    account again.

    3) After you receive the error, go to Menu Help -> Tools -> Examine, and look for the

    value for field CHARGE_ACC_WF_ITEMKEY (Click on the arrow next to the Field Zone

  • 8/8/2019 ERP Support Tips


    and then Query up the CHARGE_ACC_WF_ITEMKEY). Note down the itemkey value

    required to run wfstatus.sql. (NOTE: If the value for the field is _SYNCH#_, the profile

    option Account Generator: Run in Debug mode has not been set to Yes).

    4) After getting the itemkey value, discard changes in the distribution zone, navigate

    to lines/header zone and commit (Save). This is important for the wfstatus.sql toretrieve any data.

    How to create an AR Debug log file

    This function is normally used in the Transactions, Customers and Receipts screens.

    The output gives details of the packages that are being used on the process when an

    error occurs. The Debug steps are as follows

    1) Go through the steps that you would perform to encounter the error/problem but

    stop just before you would get the actual error. i.e. Don't hit the key that will invoke

    the error.

    2) From the Help menu at the top of the form, select: Help, Tools, Examine. Enter

    the password if prompted. This will give you a box with 3 fields. Enter the Following:

    In Block: Parameter

    In Field: AR_DEBUG_FLAG

    In Value: FS Where F = File S = Screen.

    relates to the UTL_FILE_DIR directory where the database writes files. There are afew ways to determine what the UTL_FILE_DIR setting is

    3) Run the following in SQL*Plus for the related instance:

    Select value from v$parameter where upper(name) = 'UTL_FILE_DIR';

    4) The UTL_FILE_DIR setting should be held in the init.ora file under the

    $ORACLE_HOME/dbs directory. There is no default setting for this parameter if it is

    not set. If the setting needs to be added to the init.ora file, then please note thatyour DBA will need to bring down the database instance and bring it back up for the

    changes to be effective.

    is whatever you wish to call the output created. i.e. Customers.log

    5) Hit the key that will invoke the error/problem. A log file should then be created in

    the output directory with the filename you have specified.

  • 8/8/2019 ERP Support Tips


    ORA-01000 Maximum Number Of Cursors Exceeded

    This is one of the most common errors that can occur. The reason for this error is that

    Oracle has reached the set limit of open cursors allowed for that executable or User

    session. It is the initialisation parameter OPEN_CURSORS that determines themaximum number of cursors. If it is a custom program that fails with this error, then

    a developer should review the code to see if the program can be modified to use

    fewer cursors. If the error occurs often within your Oracle Applications, your DBA

    should shut down Oracle, modify the OPEN_CURSORS parameter to increase its value,

    then restart the database.

    ORA-01562:failed to extend rollback segment number (n)

    ORA-01628:max # extents (n) reached for rollback segment

    Errors with rollbacks segments are common in the Oracle database when storage

    becomes a problem. If you encounter errors with rollback segments, or a max #

    extents error referring to a database object such as a table or index, contact your

    DBA, as the size of the rollback segment will need to be increased, or the value of the

    maximum extents parameter increased to allow for further growth.

    REP-1222 Invalid body size

    REP-1212 Object is not fully enclosed by its enclosing object

    REP-1211 Object is larger than enclosing bodyThese are common errors that can occur with the running of reports. The cause of

    these errors is usually that the Rows/Columns defined in the Printer Style you are

    trying to use do not cover the whole of the report. If it is a Standard applications

    report, then the solution is to increase the Col/Row for the Printer style. If it is a

    customised report you need to establish the size of the report and set-up a Printer

    Style as appropriate.


    If your problem is in fact caused by data corruption, a data fix will be required to

    update the records in error. If you need to run a data fix script, it is essential that you

    perform a backup of the Oracle Application tables that will be affected by the fix.

    Additionally, to form an effective audit trail, output should always be spooled before

    and after the data fix to verify its results. Always review any data fix scripts sent to

  • 8/8/2019 ERP Support Tips


    you by Oracle Support Services before you run them. Back up the tables the script is

    either inserting, updating or deleting records from, and if the script does not contain

    a SPOOL command, then update the script to insert the SPOOL command before

    execution of the script, and insert a SPOOL OFF command after the last SQL

    statement. Example as follows


    Update ap_payment_distributions_all

    set amount = base_amount

    where invoice_payment_id = 13548

    and payment_line_number = 33;


    Last, but not least, always run the data fix script in a Test environment before

    running it against Production. It is always good practice to perform regular refreshes

    of your Test environment to keep your Test environment up to date with your

    Production environment.

  • 8/8/2019 ERP Support Tips



    Depending on the Oracle Applications version you are running, the architecture of the

    Oracle Applications environment will vary. The fundamental differences in the

    architecture are as follows -

    Release 10

    The Release 10 architecture supports both the Graphical User Interface (GUI) Client

    and Character Mode forms. Release 10 SmartClient (SC) works with an Oracle

    Applications server running Release 10.7. Through the character mode forms tool, you

    can access data through workstation clients. There are generally two tiers in this


    Database tier (contains the database, application logic and the concurrent mangers)

    Client tier (contains logic required to display the GUI front-end forms and some of the

    business logic and error handling)

    (Source: Oracle Applications for MS Windows Clients Installation Manual. Applications

    Server for Window NT Edition. Release 10SC Production 16.1 pg 2-2)

  • 8/8/2019 ERP Support Tips


    Release 10.7 NCA Release 11

    A third tier is introduced in the later releases of Oracle Applications. The client

    interface is provided through a Java Applet Viewer or Web browser that launches the

    applications forms server from the middle tier. The three-tier structure is as follows Database tier contains the data manipulation logic and manages the database.

    Application tier contains the applications specific logic and manages the Oracle

    Applications and other development tools (the forms and procedures that define the

    application), including the Web Application Server software.

    Desktop client tier provides the User interface display.

    The introduction of the middle Application tier eliminates the need to install and

    maintain application software on each desktop client, and helps to reduce network

    traffic. The fundamental difference with the Internet Computing architecture is that

    only the presentation layer of Oracle Applications is on the desktop tier in the form of

    a plug-in to a standard Internet Browser.

    (Source: Oracle Applications Release 11 for Unix Concepts pg 1-2)

    Release 11i (11.5)

    The Internet Computing architecture of Release 11i provides a framework for multi-

    tiered, distributed computing that supports Oracle Applications products. It is the

    100% Internet Oracle Applications suite that allows companies to run their worldwide

    operations from a single, centrally managed site. The Database tier manages an

    Oracle8i (8.1.6) Relational Database Management System (RDBMS) which offers

    additional features only used by this Release.

    (Source: White Paper - Upgrading to and administering Release 11i: A Technical


  • 8/8/2019 ERP Support Tips


    2. Basic Structure of the $APPL_TOP

    It is important to have a general understanding of Oracles APPL_TOP and the

    directory structure of the Oracle Applications product files. It helps to know where

    the applications code resides for the various products, as some of the tools andtechniques for investigating support issues require you to directly access these

    specific program files.

    In the basic directory structure, there is a top Applications directory that holds one

    set of product files. You create this directory and set the environment variable

    $APPL_TOP to point to it. The Autoinstall process sets up a directory tree for each

    fully installed or dependent product within this directory.

    A product directory tree starts with a directory that uses the products abbreviation,

    eg ap for Oracle Payables, gl for Oracle General Ledger. Below that is a sub-directory

    that uses the product version. This is known as the Product Top directory eg

    $GL_TOP, $FA_TOP, $AP_TOP. Beneath each Product top directory are various sub-

    directories to hold the different types of product files. Forms programs will reside in

    the forms directory, report programs will reside in the reports directory, sql scripts in

    the sql directory and so on.

    Custom programs should reside in a custom directory structure under the customapplication top directory.

    A Look at $APPL_TOP

    The Oracle Application software is installed in one common area in a subdirectory

    tree fashion. The top directory for this repository is defined as $APPL_TOP. All other

    subdirectories will key off of $APPL_TOP. For each product module that you haveinstalled, there will be another $_TOP variable defined. For example, if you have

    Accounts Payable, General Ledger, and Purchasing, then you will have a $AP_TOP,

    $GL_TOP, and $PO_TOP. Underneath these directories, you will notice a common

    directory tree including /bin, /forms, /lib, /sql, /install, /srw, etc. Each directory

    has its own unique use, and is similar for every product top. You should become

    familiar with these standards and the contents of these directories.

  • 8/8/2019 ERP Support Tips


    Note that there exists other "tops" not associated with the product modules that your

    site may use. These are either "shared" products or supporting products that Oracle

    installs for you. We will explain a few of them in more detail. Note the file called

    APPLSYS.env under $APPL_TOP. This is one of the environment files that are sourcedin to define your environment variables and directories necessary for your

    applications. It will define $APPL_TOP, your product tops, and other variables. We

    will discuss some of the "output" directory variables in a moment.

    A couple of other directories worth noting under $APPL_TOP include

    $APPL_TOP/install and $APPL_TOP/patch. The /install subdirectory holds many of the

    utilities used for the initial installation. It also has a /log subdirectory where

    installation, patches, and other adadmin utility log files will go. You will visit this

    directory often. Typically, you will have $APPL_TOP/patch for holding patches that

    Oracle Support may send you (will send you). You can put your patches in another

    directory, but this is the default. You will become very familiar with patches, too.

    A Look at $FND_TOP

    Another top that you will find is $FND_TOP. When Oracle started writing the financial

    applications software, they developed a core set of utilities and objects called

    Application Object Library (AOL). From these objects, they wrote the "foundation" forthe Oracle Applications, referred to as Application Foundation. This foundation code

    is stored under $FND_TOP. As product modules developed, they hooked these into the

    Application Foundation infrastructure utilizing the AOL objects. Examples of these

    AOL objects and FND products include the concurrent managers, quick picks, zooms,


    Notice that $FND_TOP has a very similar directory tree to the other product modules.

    You may have noticed that when you fire up the applications, that you call the scriptfile "found" (short for Application Foundation) which executes the $FND_TOP/bin/aiap

    executable and passes the username/password stored in the variable $GWYUID

    (typically applsyspub/pub) to get you to your initial login screen. See, its not magic --

    just code. Many of the topics that we will cover, especially the Concurrent Managers,

    are found in $FND_TOP.

  • 8/8/2019 ERP Support Tips


    A Look at $AD_TOP

    Most of the other utilities used by the Oracle Fin-Apps DBA and which we will discuss

    in detail are found in $AD_TOP. Of particular interest are the /bin and /sql

    subdirectories. You will find the following executables in $AD_TOP/bin: adaimgr(autoinstall for the installation or upgrade of the software), adpatch for administering

    patches, and adadmin which is a menu driven utility for maintaining both the Oracle

    Applications database and software. Many of these utilities in turn call other $AD_TOP


    Output Directories of the Oracle Applications

    There are several directories where output is written. These directories require

    routine cleanup and maintenance. The jobs that are run from the concurrent

    managers create both log and output files (reports). The location of these files

    depends upon the variables $APPLCSF, $APPLLOG, $APPLOUT, and $*TMP. The

    $APPLLOG and $APPLOUT variables are typically set to "/log" and "/out", respectively,

    but they can be set to other values. The location of these subdirectories depends

    upon the value of $APPLCSF.

    If $APPLCSF is set to a directory, then all of the product modules (AR, AP, PO, etc.)

    output will go to a common "log" or "out" directory. The typical setting, though, is tonot have $APPLCSF set to any value. When this is true, then the output for the

    product modules defaults to the specified "log" and "out" directories under the

    corresponding product module top. For example, concurrent manager jobs run from

    an Account Receivables responsibility would find the logs and data output in

    $AR_TOP/log and $AR_TOP/out.

    I would advise you to not set $APPLCSF. This way, you can more easily find and

    categorize your output. There is generally a lot of output, anyway, and you can stressthe inodes by having too many files. Be sure that your temporary directories, such as

    $APPLTMP or $REPTMP get cleaned up and don't fill up a file system. Note that any

    SYSADMIN responsibility output will go in $FND_TOP/log or $FND_TOP/out.

  • 8/8/2019 ERP Support Tips


    3. The Concurrent Managers

    One of the most attractive features of the Oracle Application Software is the

    concurrent manager. Basically, this is a batch job scheduling system for running

    reports and background tasks. From the concurrent managers you can manage queues,

  • 8/8/2019 ERP Support Tips


    workshifts, access and security, job priorities, job log and output, job automation,

    and assorted job compatibility (or incompatibility) rules.

    This feature is one of the key areas that can consume much of the Oracle Fin-Apps

    DBA/SYSADMIN time. To find more complete instructions on how to setup and use theconcurrent managers and the jobs that they run, refer to the AOL Reference Manual.

    For the purposes of this presentation, we will discuss major concepts in setting up the

    managers, performance issues, and other general tips and suggestions.

    Basic Tuning of the Concurrent Manager

    We go back to the age old concepts of computer tuning and balance loading for OLTP

    versus Batch Processing. OLTP (on-line transaction processing, or "real-time"

    computing) is where you have end-users doing their work on the screen needing quick,

    real-time results -- especially if they are servicing clients in person or on the phone.

    These requests need to be completed as soon as possible as to not disrupt the

    business and revenue flow! An example of these transactions may be your Order Entry

    people in customer services.

    Note: Just because an on-line transaction submits a job to the concurrent manager (or

    the "background") that this does not necessarily qualify it as a "batch-processing" job.

    On the other hand, batch-type jobs can afford to be completed at a later time than

    when initially entered. They usually can be grouped together (batched) and processed

    outside of the normal business hours. Examples of these type of reports could be

    financial reports, summary reports, end-of-day processing, etc. Some jobs are

    required to assist the on-line transaction processing but can be batched (like a sales

    forecast or open ticket report) but needs to be completed prior to the days activities,

    rather than after.

    You may be in a 7x24 shop where OLTP is always a priority. Balancing your OLTP

    versus batch jobs may be a little more complicated. Still, your objective is to reduce

    the impact of the non-critical resource hungry jobs against the OLTP transactions. The

    batch jobs will just have to work when OLTP demands drop. You do this by managing

    queues, workshifts, priorities, incompatibility rules, and . . . end-user training or


  • 8/8/2019 ERP Support Tips


  • 8/8/2019 ERP Support Tips


    Example script for starting the managers:

    dateecho "Executing script ..."

    echo "Starting Concurrent Managers ..."

    startmgr sysmgr="apps/fnd" mgrname=prd sleep=20


    Actually, I would advise you to use symbolic parameters for the APPS password

    instead of hard coding it. The "sleep" parameter tells the internal manager to search

    fnd_requests every 20 seconds for new requests, rather than the 60 second default.

    The internal log file will be called prd.mgr (typically found in $FND_TOP/log). There

    are other parameters available, too, such as the debug option. Consult your manual

    for more details.

    Example script for stopping the managers:


    echo 'Stopping Concurrent Managers ...'#The following is one command line

    $FND_TOP/bin/CONCSUB apps/fnd SYSADMIN 'System Administrator' SYSADMIN WAIT=Y


    #End of command line

    ps -ef | grep LIBR


    echo 'Concurrent Managers Stopped'

    exit 0

    Notice that does not run a command line executable to directly stop the

    managers. Instead, it submits a concurrent job via the concsub utility. The WAIT

    parameter tells the job not to process any further until all the managers have

    shutdown before proceeding and eventually exiting the script.

  • 8/8/2019 ERP Support Tips


    Debugging Concurrent Manager Errors

    Look for errors in the logs. The internal manager's log file will usually be in

    $FND_TOP/log (see previous discussion on defining log and out directories) defaulting

    to std.mgr or named as you specified in the command line parameter, mgrname=. Theinternal manager monitors the other queue managers. You will see the startup,

    shutdown, print requests, and other information in this log. You may also find errors

    as to why the internal or subsequent slave managers could not start.

    All of the other managers have dedicated logs, too. They are preceded with a "w" or

    "t" followed by an identity number, such as w139763.mgr. Each queue will have one of

    these log files. You can see individual jobs and associated request ids in each of these

    files. You can review error messages, too. Occasionally, a job will fail and take the

    manager down with it. The internal manager will sense that the queue is down and

    restart it on the next pmon cycle.

    Suggestion: We will discuss purging of the fnd_concurrent_request table and

    associated log and output files, later, but I would make this suggestion. Purge these

    manager files frequently (daily) so that you can easily perform a search on "error"

    when trying to debug concurrent manager errors.

    Kick Starting Dead Managers

    Sometimes you may encounter difficulty in starting either the internal concurrent

    manager or the other slave queues. Consult the log files for error messages and take

    appropriate action to resolve the problem. If you are unsuccessful, then enter the

    "verify" command in the concurrent manager screen to force the internal manager to

    read and initiate the target number of queues specified. If that doesn't work, try to

    deactivate or terminate the managers, then restart them. If you have trouble bringing

    them down, you may have to perform a "kill" on the background process. You canidentify the managers with "ps -ef|grep LIBR" command. If you still encounter

    problems, make sure that there aren't any processes still tied to the managers. If you

    find any, kill them.

    If you still encounter problems, then the statuses are probably improperly set in the

    tables. For example: You may see the error in the internal std.mgr log file stating

  • 8/8/2019 ERP Support Tips


    that it was unable to start because it has already started! You have verified that

    there are no "FNDLIBR" background processes. The problem is that the tables have

    improper statuses. You will have to clean up these tables. Here are some queries. I

    put them into scripts and keep them handy for when the time arises because the

    statuses are not that easy to remember.

    Reset the concurrent queues:

    UPDATE fnd_concurrent_queues

    SET running_processes=0, max_processes=0;

    Remove any completed jobs: (optional)

    DELETE FROM fnd_concurrent_requests

    WHERE conc_process_status_code='C';

    Set jobs with a status of Terminated to Completed with Error: (optional)

    UPDATE fnd_concurrent_requests

    SET status_code='E',phase_code='C'

    WHERE status_code='T';

    Delete any current processes:

    DELETE FROM fnd_concurrent_processes;

    I have listed these in descending order of frequency that I have had to use them.

    There is a paper available from Oracle Support which describes these and more.

    Purging Concurrent Manager Logs and Output

    The concurrent managers create several table entries and file output in the /log and

    /out directories. You should purge these frequently to reduce excessive table growth

    and fragmentation, and avoid performance degradation of the concurrent manager

    processes. You should also decrease the used space on your disks from old log and

    report files. This will also relieve stress on the inodes from a large number of files.

    Under SYSADMIN, setup a reoccurring report called "Purge Concurrent Request and/orManager Data". There are several parameters, but I typically try to setup two jobs. 1)

    One job for "Manager" data -- that's the concurrent manager log files typically found in

    $FND_TOP/log. I set the frequency to daily, and have it purge down to one day. 2)

    Another job for the "Request" data -- this is for all other modules outside of the

    SysAdmin responsibility, such as AR, PO, GL, etc. I typically try to keep only one

    week's worth of data out there on the system. Your needs and capacity may vary, so

  • 8/8/2019 ERP Support Tips


    set accordingly.

    This purge process does two things: 1) Deletes rows from the fnd_concurrent_requests

    tables, and 2) Deletes both the log and output files from the associated $XX_TOP/log

    or /out directories. If for any reason the file delete did not complete, but the table

    data was purged, then you will need to manually purge the output files from the /logand /out directories. This can happen if the privileges were incorrectly set, or you

    replicated a copy of the production database to your development environment, or

    the file system was not mounted, etc.

    Purge Signon Audit Data

    This is another purge report, like above. Only this purges the signon audit data which

    records every login to the Oracle Applications. Set the frequency and retention equal

    to that of your request data purge.

    Performance Tuning of Concurrent Manager Jobs

    What has been described thus far is balancing job throughput. Yet, the jobs

    themselves may be in need of sql tuning or resolving problems in the database. We

    won't go into detail of sql tuning -- that is a typical skill set that should be handled by

    the IT staff. What I want to discuss here are ways of identifying and classifying

    problems within the Oracle Applications.

    FND Tables Can Speak Volumes

    The concurrent manager is just a scheduling system that keeps track of jobs,

    parameters, scheduling information, and completion statuses of every job submitted.

    By querying these tables, you can learn much about the patterns of your site,

    including performance trends.

    I strongly suggest that you become familiar with these tables and develop reports

    against these tables. Some of the most useful tables are the fnd_concurrent_% tables.

    Things to look for are which jobs are run, how many times executed, completion

    status (especially "errors"), and run times for these jobs.

    Where Can I Get Help?

  • 8/8/2019 ERP Support Tips


    When it comes to looking for established help on tuning your concurrent manager

    jobs, there is an excellent reference that can never be exploited enough... the white

    paper on Managing the Concurrent Managers or ("How to Herd Cats") by Barbara

    Matthews. See proceeding papers from the OAUG Fall 1997 convention. Thispresentation has been very useful to me. I have modified several of these scripts to

    my clients' needs.

    My favorites are daily errors, daily and weekly hogs, the min/max reports, and the job

    schedule report (note that these are not the exact names that you'll find). Here are

    some ideas on how to use these reports.

    The daily errors report shows me every job that completed with an error status. I

    review these from time to time to look for trends. The error could be caused by a bug

    (so then you open a tar and look for an existing patch). The problem is usually

    attributed to user error, such as bad parameter input. But don't let the error go on --

    it could be an indication that the user needs some training or other help (you'll know

    the user name because it provides the request id number that allows you to view all

    the details and log of the job -- if you haven't purged it, yet).

    The hog reports flag every job that exceeds some set time threshold (such as 20

    minutes). It also sets a submission time range, such as weekdays 6:00 AM to 6:00 PM.The idea here is that we are looking for jobs with very lengthy completion times

    running during standard operating business hours (the prime OLTP window). If a job

    exceeds this limit, then it is taking resources away from your OLTP users and should

    either be 1) tuned to reduce execution time, or 2) moved to the "batch" processing

    window or queue during the off-hours.

    Before you tune a "hog", I would suggest that you see if a performance patch has been

    issued on this program. Many times there is, and this can save you the trouble oftuning it -- and crossing that dilemma of introducing a customized piece of code into

    your environment.

    The min/max reports can be modified to sort the jobs in ascending or descending

    order based upon the execution time or number of times executed. This report takes

    some interpretative skills. For example, lets say that you identify the job that has the

  • 8/8/2019 ERP Support Tips


    longest execution time... say 4 hours! At first glance, this looks like a sql tuning

    candidate. A closer look, though, reveals that the minimum time it took to run the

    job was only 2 minutes -- and that the average time for 300 submissions in one day

    was only 5 minutes! Now, what you have is some sort of exception. You should cross-

    reference this job to the "hogs" report -- it should be there. Or, see if it was in theerrors. By finding the request id of this aberrant job you can review the details. You

    may find that the parameters specified a much larger data set, or was incorrect, or

    many other things.

    If you finally determine that the job was correctly submitted and that the rest of the

    evidence points to an optimized sql code set, then you have probably encountered a

    "non compatible" job! In other words, the job is fine by itself, but may suffer

    drastically due to contention with other jobs run at the same time. With more

    detective work, you should strive to find which jobs it is incompatible with and

    rearrange queues, priorities, or compatibility rules to ensure that they will not run


    The job schedule report shows all the scheduled jobs that keep submitting

    themselves, automatically. There are a few things I look for, here. One is the sheer

    volume of jobs that may be scheduled -- are they really needed? Often these jobs get

    scheduled, then forgotten, and are no longer useful. Or is it a batch oriented job that

    runs during peak time that should be rescheduled to a more practical time slot? Or isthe owner of the job still an employee? I have seen many "ghost" jobs that were once

    submitted by users who have left the company -- but their reports still run,


    One last item about scheduled jobs. See if the jobs are overlapping themselves. When

    specifying the resubmission parameters, you can have a job start at a fixed time, or

    reschedule at a time interval calculated when the jobs starts, or reschedule at a time

    interval after the job completes. I often find jobs scheduled to resubmit some timeafter the first job starts, like every 15 minutes. Maybe the job used to complete in 5

    minutes. Yet, as the database grows, the job may now be taking more than 15

    minutes to complete. Hence, it submits the same job, again, when the first one hasn't

    even completed, yet! Then this can cause contention degrading the performance time

    of both jobs and the cycle repeats itself and degrades further and further. I would

    suggest that you schedule jobs to resubmit themselves on a time delay after the

  • 8/8/2019 ERP Support Tips


    previous job completes!

    I Didn't Know Those Scripts Were There!

    There are some other existing scripts which may be of benefit to you, but I must firstput in a very strong disclaimer: CAUTION: Do not blindly run these scripts without

    analyzing their purpose, impact, and possibly consulting with Oracle Support! Test

    them in your development environment, first.

    I must confess that I do not fully understand why all these files are here. I suspect

    that many are used in the installation/upgrade and use of the applications. I have not

    found deliberate documentation of these scripts, other than what I can see in some of

    the script text. Yet, I have used some of these scripts to great satisfaction -- or at

    least to learn about the usage of certain tables. These scripts are in $FND_TOP/sql.

    The ones of interest for the concurrent managers are afcm*.sql and afrq*.sql. These

    range from reports on the concurrent managers, locks, gridlock, etc. You can also find

    useful scripts in $AD_TOP/sql, too. Again, BE CAREFUL!

    Things to Avoid Regarding the Concurrent Managers

    These following tips seem to be common sense, but I am still amazed at how often I

    see these abuses and misunderstandings, so I will mention them...

    Use of the Reprint Option: Do not allow your users to run jobs multiple times in order

    to recreate the same output. They can view it offline or do a reprint on a previously

    run job. There are other third party tools, too, that give more flexibility in viewing

    and formatting the outputs, too.

    Use Query Enter to Find Your Jobs: If a user cannot see their job on the immediate

    screen, then scroll down or enter a query to further define the job that they arelooking for. I have seen sites where the user couldn't find the job they submitted on

    the first screen, so they would submit it again!

    Whoa! on the Refresh Screen: It is very, very common to have your whole company

    just hitting that refresh key on the concurrent request screen in an effort to see their

    job go into the queue or its completion status -- especially when performance is

    already suffering! But this only contributes to the problem! This is one of the most

  • 8/8/2019 ERP Support Tips


    common queries possible. For one, the internal manager scans this table at whatever

    the pmon interval (the concurrent manager pmon, not to be confused with the Oracle

    background pmon process) where it scans the fnd_requests table for the next set of

    pending jobs to be processed.

    Discourage Multiple User Logins: Multiple logins by the same user to get more work

    done is often contributing trouble to an over researched system. Sometimes this is

    unavoidable because the user wears different "functional" hats and must view

    different screens/data within multiple responsibilities. Some also find it annoying to

    login and navigate to particular work screens, and then keep idle sessions active until

    they need them. Try to educate your users that they consume resources (memory,

    CPU, etc.) every time that they do this. In the newer NCA versions, navigating to

    different screens and responsibility areas will be made easier via shortcuts and should

    help to eliminate this abuse.

    Eliminate Redundancy of Similar Jobs: Users often submit the same job multiple times

    in the same time frame, distinguished only with minor changes to the parameters.

    These jobs hit the same tables over and over again and can even create locks and

    resource conflicts among themselves. Many times they would find the overall

    throughput to be better if they single threaded the jobs one after the other. This can

    be managed by user education or by the SYSADMIN single threading the queue or

    placing incompatibility rules that limit the same program to run with itself.

    Another variation of this problem is having different users running the same or similar

    jobs at the same time. It may be better for the SYSADMIN to schedule these jobs to

    resubmit themselves in the concurrent manager at predetermined intervals and take

    away the ability for the end-users to submit the jobs, themselves. This should reduce

    the frequency and burden on the system, yet allow the users to still have the jobs and

    processes run in a timely manner for their use.

  • 8/8/2019 ERP Support Tips


    4. Utilities for Maintaining the Applications

    There are many tools and utilities available to you for maintaining and upgrading your

    applications. Some are well documented, others are more mysterious. I'll describe

    some of the major utilities. Note that most of these utilities are in $AD_TOP/bin or


    Patching with the Adpatch Utility

    This utility is used for applying patches that you receive from Oracle Support. When

    you uncompress the patch from Oracle, you will get at least one driver file

    (patch.drv), a readme.txt, and new code to patch your applications. The patch.drv

    file is read by the adpatch utility and performs a multitude of tasks. It basically

    checks the versions of your code to make sure that the patch code is more recent,

  • 8/8/2019 ERP Support Tips


    moves the new code to the proper directories while making a copy of the original

    suffixed with an "O", updates the library file, links object code to make new

    executables, compiles or generates code, and logs all of its activities.

    All of these tasks are performed by other utilities in the $AD_TOP/bin directory,including adlib*, admvcode, adrepgen, adrelink, adfrmgen, etc. Look at the log file

    for your adpatch task and you will see the utilities that were called. These utilities

    will match up to the operative key words in the patch.drv file.

    You should ALWAYS review the readme.txt file prior to applying a patch. You need to

    verify that the patch is going to do what you intended, and see if there are any other

    manual tasks to perform either before or after applying the patch. If sql scripts are to

    be performed, the patch.drv usually moves the sql script to the directory but does not

    execute it. The readme.txt file will direct you to run adpatch again, and direct you to

    specify the db*.drv file as the patch input. This will execute the sql scripts.

    Patching Suggestions and Tips

    Always make a backup of the directories that will be affected prior to applying a

    patch -- a patch can be a very nasty thing to rollback! Even though admvcode will

    make a backup copy of most files suffixed with a capital "O", it is not very reliable in

    rolling out a patch. This is because sometimes patches are "bundled" up with otherpatches and the affected files may be patched multiple times. Hence, the backup file

    "O" may actually be a backup of the backup! Without your own backup, you cannot

    rollback to the original.

    Regardless of what the readme.txt file says, to really be certain which files and

    activities can occur, look at the patch.drv and db*.drv files. If it isn't in the patch

    driver file, then it isn't going to happen.

    When prompted for the patch log file, do not take the default name "adpatch.log". I

    recommend that you use the same patch/bug number, such as .log. This enables you

    to quickly review the results of your patch without stumbling through reams of

    previous patches.

    A running log of patches applied resides in $APPL_TOP/applptch.txt. DO NOT DELETE

  • 8/8/2019 ERP Support Tips


    THIS FILE! It is invaluable when determining which patches have been applied, when,

    and what actually happened in the patch. I am finding this file even more critical in

    considering the NCA upgrades and possible Y2K upgrades -- operations where you may

    have to lay down a new baseline of the applications and reapply your patches to

    recreate your current configuration!

    To learn more about the patching process and several other utilities, investigate the

    log and patch.drv files. Many of these utilities can be run by themselves. You may find

    use (at least understanding) of these utilities.

    Adadmin Utilities

    This is an interactive menu available to you to maintain several aspects of your Oracle

    Applications environment. The menu divides into two categories: Database and File


    The database screen gives you options for creating or maintaining database

    structures, data, or privileges. Most of these activities are encountered during

    installation or upgrades. You