-->
Thread Pool count will give you the idea about the WebLogic Server instance throughput. First let us see how to monitor a server instance with WLST. if you provide the instance name the script will search the corresponding listen address, listen port for that instances then forms a URL using t3 protocal(WebLogic specific protocol) which is used to connect the instance and get the serverRuntime MBean which will contains that servers ThreadPoolRuntime.
-->
Hogging Threads
Hogging Threads that have taken the too much time and we can assume that they are never going to come back. Hogging threads help us take some decisions, lets say many threads are hogging, we may take a decision to create new threads for next cycle.
My understanding about Thread States in WebLogic Server:
- ACTIVE
- STUCK
- STANDBY
Threads might wait for other thread to release resource. This might happen due to application varialbles. The variables are 2 types thread-safe other is risk for thread. All local variables in the methods are thread-safe. The variable defined in class level are unsafe. which causes memory leak, this state of threads are known as hogging. WebLogic identify a thread as hog by the time interval. If thread is waiting more than 600 sec will be treated as hog. STUCKthread interval we can tune as per the project need.
If the number of HoggingThreadCount increases then the server health is in dangerous. That time you can take the ThreadDump
After Threads increase to a max utilization then the thread will be in STANDBY state.
Let us see the following script will get the Thread statistics for a given Weblogic server instance.
# This script is for single instance thread statistics
# You can enhance it further to take thread dump as per your env.
import sys
from java.util import Date
ucf='scriptpath/userConfigFile.sec'
ukf='scriptpath/userKeyFile.sec'
ECODE='\n \033[0m' # ending of color code
def ThreadCnt():
try:
print 'Connecting to Admin server....'
connect(userConfigFile=ucf, userKeyFile=ukf, url='t3://admindns:port')
except:
print 'Admin Server NOT in RUNNING state....'
urldict={}
serverlist=getRunningServerNames() # Getting Serverlist
for svr in serverlist:
cd("/Servers/"+svr.getName())
urldict[svr.getName()]='t3://'+get('ListenAddress')+':'+str(get('ListenPort'))
x = raw_input('Enter a server instance name : ')
try:
connect(userConfigFile=ucf, userKeyFile=ukf,url=urldict[x])
serverRuntime()
openSocks = cmo.getOpenSocketsCurrentCount();
print('Open Sockets:: ' + str(openSocks));
cd('serverRuntime:/ThreadPoolRuntime/ThreadPoolRuntime/')
compReq = cmo.getCompletedRequestCount()
status = cmo.getHealthState()
hoggingThreads = cmo.getHoggingThreadCount()
totalThreads = cmo.getExecuteThreadTotalCount()
idleThrds = cmo.getExecuteThreadIdleCount()
pending = cmo.getPendingUserRequestCount()
qLen = cmo.getQueueLength()
thruput = cmo.getThroughput()
if idleThrds == 0:
pstr='\033[1;47;31m' # RED color
else:
pstr='\033[1;40;32m' # GREEN color
print(pstr+'Status of the Server: ' + str(status) +ECODE
+'The completed Requests: ' + str(compReq) +ECODE'
+'Total the threads no s: ' + str(totalThreads)+ECODE
+'The Idle threads: ' + str(idleThrds)+ECODE
+'Hogging threads : ' + str(hoggingThreads)+ECODE
+'Pending : ' + str(pending)+ECODE
+'ThreadPool QueueLength: ' + str(qLen)+ECODE
+'Server (Throughput): ' +str(thruput)+ECODE)
except:
print 'Exception... Unable to connect to given Server', x
pass
quit()
def quit():
d = Date() # now
print d
print '\033[1;40;32mHit any key to Re-RUN this script ...'+ECODE
Ans = raw_input("Are you sure Quit from WLST... (y/n)")
if (Ans == 'y'):
disconnect()
stopRedirect()
exit()
else:
ThreadCnt()
def getRunningServerNames():
domainConfig()
return cmo.getServers()
if __name__== "main":
redirect('./logs/ThreadCntwlst.log', 'false')
ThreadCnt()
print 'done'
WebLogic Server Health Status can be one of the following:
OK is indicates everything fine, no worries!!
WARN raised when there is few stuck threads in the server instance.
LOW_MEMORY_REASON is going to tell you about JVM crash expected. You can configure to 'Exit' the managed server on low memory conditions with the help of NodeManager and WorkManager.
CRITICAL when multiple number of stuck threads happening and the threadpool count reaching unsual number. This case you need to suspect Network, JDBC or back-end connectivity has trouble.
FAILED happen when the new deployments fails. The NodeManager should not restart this managed server.
OVERLOADED Change the server health state to OVERLOADED on overload. The Nodemanager need to work at this state and bounce such WebLogic instance. This is a new feature of WebLogic 9.x and later versions, for detecting, avoiding and recovering from an overload condition of a WebLogic managed server. Overload protection can be used to throttle Work Managers and thread pools for performance. You can configure Shutdown the Work Manager or application on stuck threads when it crosss more than 5 or you can set threshold.
- HEALTH_OK
- HEALTH_WARN
- HEALTH_FAILED
- HEALTH_CRITICAL
- LOW_MEMORY_REASON
- HEALTH_OVERLOADED
WARN raised when there is few stuck threads in the server instance.
LOW_MEMORY_REASON is going to tell you about JVM crash expected. You can configure to 'Exit' the managed server on low memory conditions with the help of NodeManager and WorkManager.
CRITICAL when multiple number of stuck threads happening and the threadpool count reaching unsual number. This case you need to suspect Network, JDBC or back-end connectivity has trouble.
FAILED happen when the new deployments fails. The NodeManager should not restart this managed server.
OVERLOADED Change the server health state to OVERLOADED on overload. The Nodemanager need to work at this state and bounce such WebLogic instance. This is a new feature of WebLogic 9.x and later versions, for detecting, avoiding and recovering from an overload condition of a WebLogic managed server. Overload protection can be used to throttle Work Managers and thread pools for performance. You can configure Shutdown the Work Manager or application on stuck threads when it crosss more than 5 or you can set threshold.
Good Reference links:
http://forums.oracle.com/forums/thread.jspa?threadID=683752
http://forums.oracle.com/forums/thread.jspa?threadID=683752
