The need for speed: Part 2 of 7 - Performance tuning parameters: Common Topics

What are the most important things to keep in mind when changing the settings for an FEA solver?

• Get off the hard drive
• DMP until you run out of RAM
• Parallel whatever cores you have left

An FEA solver must first create a system of linear equations which describe the FEA problem to be solved.  This is sometimes called assembling the stiffness matrix, although other matrices may be assembled, such as mass or damping.  These equations are stored in matrix form, and must then be operated on by a mathematical solver in order to arrive at the solution to the FEA problem.  In approximate order of increasing memory requirements, the typical solvers seen in structural FEA are:

• Forward Integration time step: explicit transient dynamics, typically short duration transients, such as vehicle impact
• Iterative: statics, linear and nonlinear, typically dense and blocky models, such as an engine block with mild contact nonlinearity
• Gaussian Direct Elimination, non complex: statics, linear and nonlinear, steady state or implicit transient.  One of the most common solver types
• Gaussian Direct Elimination, complex : frequency response, the steady state response of a structure when loaded by loads which all have the same forcing frequency, such as a speaker generating a single tone or vehicle NVH analyses across a frequency range
• Eigenvalue solvers, modern methods (automated multilevel substructuring): natural frequency and buckling loads, also used for modal type solutions to direct frequency and transient responses
• Eigenvalue solvers, older methods (Lanczos, etc.)

Already we can see something interesting.  Whereas one might think of how 'big' their model is in terms of the number of nodes or elements, the computer sees the model as big as the solution determines it will have to be.  A model that runs quickly with particular memory settings on a particular computer when solving a linear statics problem may run much, much slower when one wants to find an associated buckling load.

So what sort of memory should we use to store the system of equations?   Given the layout of a typical computer's memory hierarchy, the most practical, high speed place to store the system of equations so that they can be quickly solved is the random access memory or memory of the computer.

At this point those with a background in computers may be wondering what if the system of equations is too large to fit in the free memory of the computer?  A typical computer program would have the operating system store whatever didn't fit into virtual memory, or hard drive space that behaves as memory.  For most programs, this is the best thing to do if there is insufficient memory.  FEA solvers, however, are different.  FEA solvers predate operating systems with virtual memory, and have developed their own specialized methods of making due with less than the ideal amount of memory.  What an FEA solver does instead is if the memory allocated to it is less than what the system of equations needs, then it will revert from what is known as an in-core solver to an out-of-core solver which will load a portion of the system of equations and solve it before putting it away in a scratch file and loading a new chunk to work on.

So now we have two interesting things to remember:

1. The same level of detail may take different amounts of memory to solve depending on solution type
2. The solver will need to know how much memory is actually available so it can intelligently decide how to store the system of equations, either in-core or out-of-core

In later posts, I will cover in more detail how to find how much memory is available in a system, find out how much memory a program estimates it will need to solve a problem, how much memory is required to quickly solve the problem, and what settings to use given the available memory and estimated memory requirements.

So now we've given a general answer to the first question, how to get off the hard drive: allocate enough memory.  Now what?

Now we DMP (distributed memory process).

Solving a large system of equations is analogous to sitting at a table and ordering a shuffled deck of cards.  If we have only one deck of cards to sort, and only one desk, and only one person sorting them, then there are no decisions to be made about how to allocate resources.  Now let's say you have a second person that can help you, and a completely separate deck of cards to sort.  If you have enough space on the desk, then they can spread out the second deck and begin sorting without slowing you down.  This is equivalent to a DMP run on a single computer: enough memory (desk space) and enough people (CPUs) and a problem that can be divided up (seperate decks of cards).  This is confusing for many at first, including myself, because a computer with more than one CPU that shares the same pool of memory is technically an SMP computer system.  So from a computer science perspective, a desk with a piece of tape down the middle is an SMP system doing DMP style work, whereas two separate desks in two separate rooms is like true DMP.

So what sort of FEA solutions lend themselves well to DMP style splitting of workload?  Many, in fact, although speedup can be very solver and solution type dependent.  DMP can be used by lanczos eigensolvers, and very easily used by direct frequency analyses, where there is a frequency that needs to be found at each frequency and each frequencies' solution can be found independently.  It can also be used by  ABAQUS for gaussian elimination, iterative, and explicit solvers, although the overhead of 'passing cards' from room to room can begin to dominate the solution times.

So what if you have only one computer, and you don't have enough memory to split the solution up any more?  This is where SMP can be of use, and is in fact one of the oldest ways that more than one CPU has been used to accelerate FEA solution times, dating to Nastran running on CRAY supercomputers.  SMP is equivalent to having only enough room on the desk to work on the one deck of cards, but more than one person reaching in and working on ordering the cards.  As you can imagine, returns quickly diminish for this type of parallelization, but if the CPUs are available there's no reason not to take advantage of it.  Nearly any solver and nearly any solution type can take advantage of this style of parallelization, with the notable exception of Abaqus, which has instead focused on parallizing in a manner that divvies up the cards intelligently ahead of time.

There is one final, more exotic, method of SMP that is sometimes available.  With the growing general abilities of the vector processors inside of the graphics cards in computers and video game systems, programmers have begun to use them as a new type of person working to sort the deck of cards.  Only in this instance, its more like a trained bag of mice, mice which still need a person to bring the deck of cards to the table and spread them out, but can handle much of the hard work from there.  This type of acceleration, oftentimes referred to as GPU acceleration, will only become more prevalent as the limits in using more and more transistors to make a single cpu faster become apparent.  As of now though, support is very limited, and only the most basic static solution of Nastran and the Gaussian solver, non complex, symmetric solver of Abaqus is supported, and only on post Windows XP operating systems.  The latest Intel chips,
Sandy Bridge and later, have a functionality similar to this that is much better supported in Nastran than GPU acceleration.

We've now covered, in very general terms, how to get off the hard drive and make the most of our ram, and having done that make the most of our CPUs and even our graphics cards.  Next we will cover solver specific tuning parameters that take advantage of these concepts.

The need for speed: Part 1 of 7 - Intro

How can I make my FEA simulation run faster?
Ask less of your computer, use more of your computer, or use a faster computer(s).

Once you've meshed the geometry, applied the material properties, applied the loads, assemble the load cases, and the model runs sometime today, doing more work to make the model run faster is probably the last thing on your mind. Tuning parameters and measures of the solution difficulty of a model can seem dauntingly complex, and the payoff from a model that runs faster may not seem worth it. However, a working knowledge of how to manage the detail and run times of a model can have some major payoffs:

• knowing ahead of time if a model has surpassed the ability of a computer to quickly solve
• minimizing storage and archiving requirements
• ability to run the same model several times to assess design sensitivities and optimize the performance of a design

This final advantage is oftentimes the most important reason to put effort into minimizing run times. The ability to analyze a design more than once in the time available can make the difference between a high performing design and and an inadequate design.

In a series of posts, I'll outline the practical steps an analyst can take to speed up their run times. This is obviously an extremely complex topic to fully cover.  Fortunately, a little knowledge can go a long way, and I will focus on the topics that typically have the largest impact.  Upcoming posts will cover:

Use more of your computer:

• Performance tuning parameters: Common Topics
• Performance tuning parameters: Abaqus and Optistruct/Radioss
• Performance tuning parameters: Nastran
• A faster computer: performance metrics

Asking less of your computer

• Reducing output requests
• Optimizing model detail and solution type

MPCs: a quick summary

Many years ago one of my coworkers built a model in Optistruct and was disappointed to see the RBE that he'd built failed to provide the restraint he expected.  He called up the technical support at Altair, and after some back and forth about the RBE element the support representative finally lost his patience: "It is not an element, it is a system of equations!"

So what are these equations, and what do they do?
First of all, they are indeed different from an element.  Although FEA software may refer to them as a type of element, they have a fundamental difference from a normal element. A normal element adds to the global stiffness, mass, and damping matrices. An MPC (multi point constraint) is more like an SPC (single point constraint), in that it modifies instead of adds to the matrices that describe the model in order to describe the special behavior of these MPCs.

So all that sounds interesting, but it hasn't really told you much about what they do. I've thought of different ways to describe them to my coworkers, and the best I've come up with is this analogy. Basically, you're including a group of grids in a type of government, and you're letting them know what kind of leader they have. So what's an RBE2? An RBE2 is a monarchy, where the control grid is king, and tells the other grids what to do. An RBE3, on the other hand, is more like a democracy, where the President does what the people tell it to do. So what works and what doesn't work when we try to make use of these MPCs?

What Works

The above works because the king can tell the dependent nodes what to do.  There are constraints on grids, but not on the same ones as the RBE2.

The above also works because the dependent grid knows what to do because the nodes that vote for it at least know what they want; if the elements beneath them were unconstrained then the whole set of elements would not know what to do.

This one works because the voters still know what they want: the constraints tell them what they want! The RBE2 above the RBE3 also knows what to do because the president is telling it what to do.  Ok, My analogy is getting old.  I'll switch to talking about independent/dependent grids.

As you can see above, I lied, the deputies are out!  Through the use of UM grids the RBE3 is not the dependent element, and can now be told what to do.

The above also works, as the UM grids allow the top RBE3 to not have its dependent grid be the same as the bottom RBE3 dependent grid.

Finally, the above works because the top RBE2 is only connected to independent grids.

You can download the above examples at RBEThingsYouCanDo.bdf

What Doesn't Work

Well this won't work, the RBE2 is trying to tell the dependent grids what to do, and the constraints are doing the same thing.

The above also doesn't work.  There are nodes which are dependent nodes in two separate RBE2's.  

The above also has problems.  The UM grids have been set, making them no longer independent, but those grids also have constraints, so again we have grids where MPCs and constraints are both trying to tell  a grid what to do.

Finally, the above is also bad.  You can't make a grid the dependent grid of more than one RBE3.

You can download the examples of things that don't work at RBEThingsYouCantDo.BDF

There are fun tricks you can do once you're feeling comfortable with RBE3s.  The two most interesting are:
-Applying load without adding stiffness.  If you have a certain total load that needs to be applied across a cross section of a model, you can use an RBE3 to apply it.  This is particularly useful at the ends of a beam.
-Constraining the overall motion of a larger body.  If it's known that something is sitting on a very soft support, such as rubber or water, an RBE3 can be created across the supported region, and through the setting of the UM card you can constrain the former dependent grid to control the overall average motion of the model.

There is, of course, something that analysts commonly do when they need to make a part of their model very stiff but don't have the time or experience to construct MPCs without errors: create elements with very very stiff properties that will behave similarly to MPCs.  The upside to this approach is that it will typically yield a very similar answer, the downside is that it may take a few iterations to find a stiffness that is stiff enough, yet not so stiff the stiffness matrix becomes ill-formed.

So wouldn't it be great if there was an MPC that automatically made a very stiff element that was just stiff enough?  Something that automatically created a perfectly rigid body but then had stiffness that connected it to the dependent grids?  Well there is in fact an element that does this.  This type of element is typically called a Lagrangian element, and it can be selected in certain load cases in Nastran and is the behavior for Kinematic (RBE2 style) and distributing (RBE3 style) elements in Abaqus.  I'll cover this topic in more detail in a later post.

Update 28 Sep 2021: Due to a Google security update the example file links expired, should be working again.

Words that are used interchangeably

FEA has a number of words and phrases that mean exactly the same thing.

Grid/Node: A point in space that defines the shape of elements.  It is also where the solution is either found or set by an SPC/Constraint/Boundary Condition.

SPC/Constraint/Boundary Condition: A point at which the solution is set to a certain value.  Examples include setting the displacement to zero at the end of a beam or setting the temperature to a certain value.  Some solvers allow some panache when setting the solution to a certain value, such as in Abaqus, where one can use the uncommon architectural term 'encastre' to describe something that is fixed in place.

Degree of Freedom, DOF, detail level: how many numbers it takes to describe a complete solution.  This is also a measure of how long it will take the model to solve.

Job/Model/Run/Input Deck:  A data file, typically ASCII plain text, which fully describes the FEA problem which the solver will find a solution to.  Sometimes a run can refer to the specific time a model was solved, in which case an analyst might describe the runs they did with different settings to find the fastest solution times based on things like memory and cpu settings.

Node/System: A single computer, one that may be a part of a networked computer cluster.  This computer will typically have more than one CPU and a single large bank of memory that can be accessed by any of the CPUs

Stiffness Matrix/System of Equations: the series of equations which describe the behavior of the FEA system, stored in matrix form, typically in a sparse symmetric storage format, most commonly one that assumes symmetric matrices.  A solver will work best if these matrices can fit into memroy without using the hard disks of the computer.

CPU/core: A single independent central processing unit.  Due to advances in shrinking chip sizes, a typically computer will typically have more than one core, packaged together, which appears to the computer no different than several independent CPUs.  High performance server computers may have more than one CPU slot, each with more than one core, for total CPU/core counts of 2x2 or more.   CPU counts of 16 are not uncommon.

RAM/Memory/Random Access Memory: Memory in a computer which is fast but loses its state when the system is turned off.  Typically measured in single digit to hundreds of gigabytes(GB).

Hard Disk/Hard Drive/SSD: Slow but permanent memory which does not lose its state when a computer is turned off.  Although SSD's have narrowed the distance between traditional magnetic memory hard drives and RAM, SSDs and hard drives remain about 1,000 times slower.

SMP/Parallel: A style of solving a system of equations that applies more CPUs but little extra memory to the same problem.  An analogy would be sorting a shuffled deck of cards on a table.  An SMP/parallel approach would be to spread the cards out over one table but have more than one person working to sort the cards

DMP: A style of solving a system of equations that applies more CPUS and more memory, either all on one node/system or several systems.  If done on a single system, it is analogous to sorting a shuffled deck of cards on a large table, where each person sorting the cards is working on their own assigned portion of the table.  If done on more than one system, it is analogous to more than one table in more than one room, which can pass cards back and forth between rooms, but always slower than back and forth across a table.

GPU/GPGPU/OPENCL: Using the simple but numerous programmable cpus of a graphics card to do general purpose vectorized computing.

Friday Night Music: Madeon and his secondary input device

I can do quality work in a number of FEA GUIs, but I am very fast in Hypermesh.  Sometimes when I'm in the zone, headphones on and focused on the model, favorite secondary input device in hand, I can feel pretty slick.  Then I see things like this and feel humbled.

What is FEA?

A hypothetical conversation with a family member unfamiliar with FEA

Your blog is... interesting?  I have no idea what you're talking about!  Why'd you make it?

A few reasons:
-I have former co-workers who still use many of the same tools that I still use, and I wanted a place where I could document some of the FEA knowledge that I've picked up in the years since we worked together
-I also wanted a place to have more informal discussions of competing commercial software packages, similar to how many consumer products are reviewed
-Finally, I wanted to play around with some fun stuff

What is FEA?  Am I saying that right, feeya?

FEA (pronounced Eff-ee-A) is a numeric technique used to simulate the structural, thermal, acoustic, and sometimes fluid behavior of a volume of material by dividing it into a series of individual elements defined by the grids that they connect to, which are the locations where the primary field variables (such as displacement or temperature) are calculated.  These field variables can then be used to estimate other variables within the elements, such as stress or heat flux.

Explain all that again, but like I'm five. Not that I don't know what you're talking about!  Ha-ha..

FEA is basically a type of computer program that lets an engineer find out if something is strong enough, stiff enough, or quiet enough.  That something could be anything, like an airplane, car, smart phone, or bell.

What's a computer?

A calculating machine made up of input and output devices which contains both working memory and a processor which operates on it.

I kid, I know what a computer is.

Ok, uhh good.

So who uses this stuff?

Lots of people!  There are dozens and dozens of FEA programs one could buy or use, and nearly any high volume consumer product  has been analyzed in FEA.  In just the last few years its use has exploded from mostly aerospace (Airplanes etc.) to almost everything else, from cars to cell phones to medical implants to diapers.

Angry Birds too right? And like movies?

Not too many games or movies have done structural physics very well.  The big exceptions have been Bridge Builder and World of Goo, both good simulations of a bridge model in FEA.  Angry birds has a collision detection engine, which tracks when things touch each other, but the behavior of a brick or piece of wood once it's been hit is actually very basic; it has a certain amount of bounce reaction and it's either damaged to failure or not.  FEA is more associated with a single piece bending and changing shape; BeamNG is the closest I've seen.

Movies are starting to become a little more sophisticated.  Pixar has been very open about their efforts to add more physics to their animation software, particularly for hair, such as the work they put into making the heroine's hair in Brave more realistic.

That was more than I really cared about there.

Yeah I know.  Fortunately I'm just imagining this conversation!

AWESOME!

Ringing in the Blog

Deformed structural models can usually be animated, depending on post-processor.  While this can be very useful, I thought it might be interesting to hear how a model sounds. I found a dimensional reference recently that allowed me to craft a very basic Abaqus Explicit model which should sound like the real thing.

Using symmetry, I was able to build the model with less than 1000 nodes.

The sound propagation is still very approximate, as sound only emanates from  a single location on the surface of the bell.  I'll follow up on this model by adding an acoustic mesh to allow pressure waves to propagate towards receiver location properly.  Still, it sounds kind of neat., and not too far off from the real thing for a first cut.