Secure Oracle Paswords for GLAST

SCCS security team has mandated that Oracle passwords be changed every six months. Before now oracle passwords at the lab have never been changed, and as a result have been commonly "baked in" to hundreds of scripts and programs.

Goals

Our goals were threefold

  1. Remove passwords baked in to existing programs and store them somewhere where they can be maintained securely.
  2. Ensure that passwords can be changed in the future without disruption of the GLAST data processing including long running servers.
  3. Avoid the necessity of sharing all the oracle passwords with all the developers in the group.

Techniques

After discussion with the SCCS database group we attempted to adopt two methodologies to address these goals:

  1. Use Oracle wallet to securely store passwords.
  2. Use Oracle roles so that developers can use their own personal accounts, and service accounts can be created specifically for use by servers and similar processes.

Status

We have succeeded in getting our tomcat and other servers to run using oracle wallet. (The tomcat servers are in production, the other servers are running in DEV and need CCB approval to move to prod). This indeed makes it possible to change the password in the database and (quickly) update the credentials stored in Oracle wallet. This took much more time than expected because of many quirks in oracle wallet, and because oracle wallet is not supported in the oracle "thin" JDBC driver we have been using up to now.

We have not so far succeeded in getting user programs (pipeline client and datacat client) to use oracle wallet. The passwords for these programs are currently stored in plain text in the script itself readable by anyone at SLAC, and in the most obvious place for a hacker to look for them. There are two reasons for needing to do this:

  • Oracle wallet is currently only usable with Oracle 10.2. The only installation of oracle 10.2 at SLAC is a 64 bit version, but SLAC has many machines (especially batch machines) still running 32 bit OS. We would need a 32 bit oracle installation to use oracle wallet for end-user programs. This needs to be checked with Ian.
  • Whether we use oracle wallet or some other mechanism we have no way to give read access only to GLAST collaborators. We need an AFS or NFS group which is maintained and contains all GLAST collaborators/users. Tom says we are working on creating an AFS group.

We have not succeeded in using oracle roles. There are currently two reasons for this:

  • All of our software currently assumes it is using a particular "default" schema. We have not figured out how to set up oracle connection aliases such that a particular default schema is set automatically when using a role account. So to use roles we would have to change all of our software which is not possible at the current time. Perhaps it is possible to do this using oracle "login scripts" but we have not had time to look into this.
  • Role accounts do not appear to allow one to modify database tables, so even if we get them working we would still need to share passwords to the owner accounts with developers. Need to double-check if this is really true?

Since we have not succeeded in getting roles working we still have to share passwords for all of our service accounts with users, and they still need to embed these passwords in their development environments. Every time the passwords are changed we need to inform developers and they will need to waste more time updating their configuration files.

In conclusion we have spent a considerable amount of time on setting up oracle wallet and experimenting with roles etc. This effort will enable us to change oracle passwords in the future without downtime for our critical servers, but has resulted in a system which is currently considerable less secure than before. This has taken a considerable amount of manpower which should have been directed to more pressing needs.

In future we should ensure that for similar security mandates

  • The SLAC security teams gives experiments sufficient lead time for such security related changes. This would typically be at least one year.
  • Discusses it detail with the experiment the need for such changes.
  • Ensure that an accurate assessment of manpower required implement changes is established at the outset, including manpower needed to research solutions before deciding on and implementing the chosen solution.
  • Where possible the manpower to implement such security changes should come at least in part from SCCS itself. The resulting solutions should be documented and made available to everyone in the lab.

Background details

Oracle wallet

Oracle wallet gives the appearance of having been written by a high-school student on their first programming assignment. In particular:

  1. It is not really a secure password store, passwords can be extracted in plain text if you have the store password. Oracle itself can bypass using the store password, but it is not entirely clear what mechanism it uses to do that, and how securely it is implemented.
  2. The options needed to create and modify credentials are not included in the documentation provided by the -help option.
  3. The program outputs bizarre messages which appear to be left over debugging print outs.
  4. When prompting for a password the program continually emits spaces in a not entirely successful attempt to hide the input password. This results in programs like putty consuming 100% of available CPU time whenever oracle prompts for a password, making it nearly impossible to enter the password. To circumvent this I had to create a script to create the wallet containing not only the wallet password but all of the oracle account passwords. Hopefully I will remember to delete this soon (but not until I am sure I won't have to recreate the wallet).
  5. The whole concept of the wallet seems to have been ill thought out (for example the need to create many aliases for the same DB connection so that each alias can use a different account).
  6. Most seriously, the password store seems to simply fail with certain aliases. For example pipeline does not work, but pipeline-ii does work as an alias. Approximately 30% of our original aliases did not work (there is no error message, oracle simply refuses to use these wallet entries when logging on and says "invalid login/password"). We wasted a lot if time trying to understand this, but failed and had to just experiment with different aliases until we got things to work, resulting in the use of strange aliases like pipelineProd. (This bug seems so bizarre I still can't really believe it, but it was totally reproducible and we can see no pattern in what works and what doesn't work).
  • No labels