Global Data Science Forum

 View Only

Hbase on Hadoop : regionserver.HRegion: Failed initialize of region= master:store ... starting to roll back memstore

  • 1.  Hbase on Hadoop : regionserver.HRegion: Failed initialize of region= master:store ... starting to roll back memstore

    Posted Mon December 20, 2021 01:59 PM

    I've run the HBase 2.3.5 on Hadoop 2.10.1 cluster for over a month. but now HBase is not started after rebooting servers.

    It seems something's wrong with WAL or mem store. but I don't know what should I do. I want to restore all my data in HBase. but I think it's ok to lose some WALs.

    Please help me. Here's the master's starting log.

    2021-12-20 19:54:26,522 INFO  [main] server.Server: Started @3044ms
    2021-12-20 19:54:26,525 INFO  [main] master.HMaster: hbase.rootdir=hdfs://challenge-cluster/hbase, hbase.cluster.distributed=true
    2021-12-20 19:54:26,550 INFO  [master/challenge01:16000:becomeActiveMaster] master.HMaster: Adding backup master ZNode /hbase/backup-masters/challenge01.e4motion.com,16000,1639997664082
    2021-12-20 19:54:26,596 INFO  [master/challenge01:16000:becomeActiveMaster] master.ActiveMasterManager: Deleting ZNode for /hbase/backup-masters/challenge01.e4motion.com,16000,1639997664082 from backup master directory
    2021-12-20 19:54:26,601 INFO  [master/challenge01:16000:becomeActiveMaster] master.ActiveMasterManager: Registered as active master=challenge01.e4motion.com,16000,1639997664082
    2021-12-20 19:54:26,603 INFO  [master/challenge01:16000] regionserver.HRegionServer: ClusterId : d86fcabd-59a8-41d0-a02c-1995baf81d26
    2021-12-20 19:54:26,604 INFO  [master/challenge01:16000:becomeActiveMaster] regionserver.ChunkCreator: Allocating data MemStoreChunkPool with chunk size 2 MB, max count 5479, initial count 0
    2021-12-20 19:54:26,607 INFO  [master/challenge01:16000:becomeActiveMaster] regionserver.ChunkCreator: Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 6088, initial count 0
    2021-12-20 19:54:26,678 INFO  [master/challenge01:16000:becomeActiveMaster] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
    2021-12-20 19:54:26,698 INFO  [master/challenge01:16000:becomeActiveMaster] coordination.SplitLogManagerCoordination: Found 0 orphan tasks and 0 rescan nodes
    2021-12-20 19:54:26,698 INFO  [master/challenge01:16000:becomeActiveMaster] hbase.ChoreService: Chore ScheduledChore name=SplitLogManager Timeout Monitor, period=1000, unit=MILLISECONDS is enabled.
    2021-12-20 19:54:26,861 INFO  [ReadOnlyZKClient-challenge01.e4motion.com:2181,challenge02.e4motion.com:2181,challenge03.e4motion.com:2181@0x65912443] zookeeper.ZooKeeper: Initiating client connection, connectString=challenge01.e4motion.com:2181,challenge02.e4motion.com:2181,challenge03.e4motion.com:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$71/249493254@4fe3fd3d
    2021-12-20 19:54:26,861 INFO  [ReadOnlyZKClient-challenge01.e4motion.com:2181,challenge02.e4motion.com:2181,challenge03.e4motion.com:2181@0x65912443] zookeeper.ClientCnxnSocket: jute.maxbuffer value is 4194304 Bytes
    2021-12-20 19:54:26,862 INFO  [ReadOnlyZKClient-challenge01.e4motion.com:2181,challenge02.e4motion.com:2181,challenge03.e4motion.com:2181@0x65912443] zookeeper.ClientCnxn: zookeeper.request.timeout value is 0. feature enabled=
    2021-12-20 19:54:26,863 INFO  [ReadOnlyZKClient-challenge01.e4motion.com:2181,challenge02.e4motion.com:2181,challenge03.e4motion.com:2181@0x65912443-SendThread(challenge01.e4motion.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server challenge01.e4motion.com/192.168.118.21:2181. Will not attempt to authenticate using SASL (unknown error)
    2021-12-20 19:54:26,863 INFO  [ReadOnlyZKClient-challenge01.e4motion.com:2181,challenge02.e4motion.com:2181,challenge03.e4motion.com:2181@0x65912443-SendThread(challenge01.e4motion.com:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.118.21:59592, server: challenge01.e4motion.com/192.168.118.21:2181
    2021-12-20 19:54:26,869 INFO  [ReadOnlyZKClient-challenge01.e4motion.com:2181,challenge02.e4motion.com:2181,challenge03.e4motion.com:2181@0x65912443-SendThread(challenge01.e4motion.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server challenge01.e4motion.com/192.168.118.21:2181, sessionid = 0x100003045ba0003, negotiated timeout = 40000
    2021-12-20 19:54:26,894 INFO  [master/challenge01:16000:becomeActiveMaster] region.MasterRegion: Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
    2021-12-20 19:54:26,897 INFO  [master/challenge01:16000:becomeActiveMaster] region.MasterRegionFlusherAndCompactor: Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000
    2021-12-20 19:54:26,918 INFO  [master/challenge01:16000:becomeActiveMaster] wal.WALFactory: Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider
    2021-12-20 19:54:26,940 INFO  [master/challenge01:16000:becomeActiveMaster] region.MasterRegion: Renamed hdfs://challenge-cluster/hbase/MasterData/WALs/challenge02.e4motion.com,16000,1639992615558 to hdfs://challenge-cluster/hbase/MasterData/WALs/challenge02.e4motion.com,16000,1639992615558-dead as it is dead
    2021-12-20 19:54:26,942 INFO  [master/challenge01:16000:becomeActiveMaster] util.RecoverLeaseFSUtils: Recover lease on dfs file hdfs://challenge-cluster/hbase/MasterData/WALs/challenge02.e4motion.com,16000,1639992615558-dead/challenge02.e4motion.com%2C16000%2C1639992615558.1639992648177
    2021-12-20 19:54:26,945 INFO  [master/challenge01:16000:becomeActiveMaster] util.RecoverLeaseFSUtils: Recovered lease, attempt=0 on file=hdfs://challenge-cluster/hbase/MasterData/WALs/challenge02.e4motion.com,16000,1639992615558-dead/challenge02.e4motion.com%2C16000%2C1639992615558.1639992648177 after 3ms
    2021-12-20 19:54:26,947 INFO  [master/challenge01:16000:becomeActiveMaster] region.MasterRegion: Renamed hdfs://challenge-cluster/hbase/MasterData/WALs/challenge02.e4motion.com,16000,1639992615558-dead/challenge02.e4motion.com%2C16000%2C1639992615558.1639992648177 to hdfs://challenge-cluster/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/challenge02.e4motion.com%2C16000%2C1639992615558.1639992648177
    2021-12-20 19:54:26,947 INFO  [master/challenge01:16000:becomeActiveMaster] region.MasterRegion: Delete empty local region wal dir hdfs://challenge-cluster/hbase/MasterData/WALs/challenge02.e4motion.com,16000,1639992615558-dead
    2021-12-20 19:54:26,969 INFO  [master/challenge01:16000:becomeActiveMaster] wal.AbstractFSWAL: WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=challenge01.e4motion.com%2C16000%2C1639997664082, suffix=, logDir=hdfs://challenge-cluster/hbase/MasterData/WALs/challenge01.e4motion.com,16000,1639997664082, archiveDir=hdfs://challenge-cluster/hbase/MasterData/oldWALs, maxLogs=10
    2021-12-20 19:54:27,027 INFO  [master/challenge01:16000:becomeActiveMaster] wal.AbstractFSWAL: New WAL /hbase/MasterData/WALs/challenge01.e4motion.com,16000,1639997664082/challenge01.e4motion.com%2C16000%2C1639997664082.1639997666976
    2021-12-20 19:54:27,139 INFO  [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false for 1595e783b53d99cd5eef43b6debb2682/proc
    2021-12-20 19:54:27,147 INFO  [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc
    2021-12-20 19:54:27,230 INFO  [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore: 1595e783b53d99cd5eef43b6debb2682/proc created,  memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE
    2021-12-20 19:54:27,240 INFO  [master/challenge01:16000:becomeActiveMaster] regionserver.HRegion: Replaying edits from hdfs://challenge-cluster/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/challenge01.e4motion.com%2C16000%2C1639987355352.1639987472287
    2021-12-20 19:54:27,272 INFO  [master/challenge01:16000:becomeActiveMaster] regionserver.HRegion: Replaying edits from hdfs://challenge-cluster/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/challenge01.e4motion.com%2C16000%2C1639987355352.1639987532323 2021-12-20 19:54:27,277 WARN  [master/challenge01:16000:becomeActiveMaster] regionserver.HRegion: Failed initialize of region= master:store,,1.1595e783b53d99cd5eef43b6debb2682., starting to roll back memstore
    java.io.EOFException: Cannot seek after EOF
            at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1648)
            at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:66)
            at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:211)
            at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173)
            at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64)
            at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
            at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:323)
            at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:305)
            at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:293)
            at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:429)
    at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:4863)
            at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:4769)
            at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:1013)
            at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:955)
            at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7497)
            at org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7455)
            at org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:269)
            at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:309)
            at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
            at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:948)
            at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2240)
            at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:621)
            at java.lang.Thread.run(Thread.java:748)
    2021-12-20 19:54:27,288 INFO  [master/challenge01:16000:becomeActiveMaster] regionserver.HRegion: Drop memstore for Store proc in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. , dropped memstoresize: [dataSize=562, getHeapSize=896, getOffHeapSize=0, getCellsCount=1 }
    2021-12-20 19:54:27,288 INFO  [master/challenge01:16000:becomeActiveMaster] regionserver.HRegion: Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682.
    2021-12-20 19:54:27,311 INFO  [master/challenge01:16000:becomeActiveMaster] regionserver.HRegion: Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682.
    2021-12-20 19:54:27,315 ERROR [master/challenge01:16000:becomeActiveMaster] master.HMaster: Failed to become active master
    java.io.EOFException: Cannot seek after EOF
            at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1648)
            at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:66)
            at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:211)
            at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173)
            at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64)
            at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
            at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:323)
            at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:305)
            at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:293)
            at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:429)
            at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:4863)
            at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:4769)
            at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:1013)
            at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:955)
            at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7497)
            at org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7455)
            at org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:269)
            at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:309)
            at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
            at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:948)
            at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2240)
            at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:621)
            at java.lang.Thread.run(Thread.java:748)
    2021-12-20 19:54:27,316 ERROR [master/challenge01:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master challenge01.e4motion.com,16000,1639997664082: Unhandled exception. Starting shutdown. *****
    java.io.EOFException: Cannot seek after EOF
            at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1648)
            at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:66)
            at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:211)
            at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173)
            at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64)
            at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
            at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:323)
            at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:305)
            at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:293)
            at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:429)
            at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:4863)
            at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:4769)
            at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:1013)
            at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:955)
            at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7497)
            at org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7455)
            at org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:269)
            at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:309)
            at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
            at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:948)
            at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2240)
            at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:621)
            at java.lang.Thread.run(Thread.java:748)
    2021-12-20 19:54:27,316 INFO  [master/challenge01:16000:becomeActiveMaster] regionserver.HRegionServer: ***** STOPPING region server 'challenge01.e4motion.com,16000,1639997664082' *****
    2021-12-20 19:54:27,316 INFO  [master/challenge01:16000:becomeActiveMaster] regionserver.HRegionServer: STOPPED: Stopped by master/challenge01:16000:becomeActiveMaster
    2021-12-20 19:54:27,317 INFO  [master/challenge01:16000] regionserver.HRegionServer: Stopping infoServer
    2021-12-20 19:54:27,326 INFO  [master/challenge01:16000] handler.ContextHandler: Stopped o.e.j.w.WebAppContext@2fdf17dc{/,null,UNAVAILABLE}{file:/mnt/data/hbase/hbase-2.3.5/hbase-webapps/master}
    2021-12-20 19:54:27,330 INFO  [master/challenge01:16000] server.AbstractConnector: Stopped ServerConnector@22a10ac6{HTTP/1.1,[http/1.1]}{0.0.0.0:16010}
    2021-12-20 19:54:27,330 INFO  [master/challenge01:16000] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@1422ac7f{/static,file:///mnt/data/hbase/hbase-2.3.5/hbase-webapps/static/,UNAVAILABLE}
    2021-12-20 19:54:27,330 INFO  [master/challenge01:16000] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@455c1d8c{/logs,file:///mnt/data/hbase/hbase-2.3.5/logs/,UNAVAILABLE}
    2021-12-20 19:54:27,331 INFO  [master/challenge01:16000] regionserver.HRegionServer: aborting server challenge01.e4motion.com,16000,1639997664082
    2021-12-20 19:54:27,331 INFO  [master/challenge01:16000] regionserver.HRegionServer: stopping server challenge01.e4motion.com,16000,1639997664082; all regions closed.
    2021-12-20 19:54:27,331 INFO  [master/challenge01:16000] hbase.ChoreService: Chore service for: master/challenge01:16000 had [] on shutdown
    2021-12-20 19:54:27,332 WARN  [master/challenge01:16000] master.ActiveMasterManager: Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null
    2021-12-20 19:54:27,333 INFO  [master/challenge01:16000] hbase.ChoreService: Chore service for: challenge01:16000.splitLogManager. had [ScheduledChore name=SplitLogManager Timeout Monitor, period=1000, unit=MILLISECONDS] on shutdown
    2021-12-20 19:54:27,333 INFO  [master/challenge01:16000] ipc.NettyRpcServer: Stopping server on /192.168.118.21:16000
    2021-12-20 19:54:27,474 INFO  [ReadOnlyZKClient-challenge01.e4motion.com:2181,challenge02.e4motion.com:2181,challenge03.e4motion.com:2181@0x65912443] zookeeper.ZooKeeper: Session: 0x100003045ba0003 closed
    2021-12-20 19:54:27,474 INFO  [ReadOnlyZKClient-challenge01.e4motion.com:2181,challenge02.e4motion.com:2181,challenge03.e4motion.com:2181@0x65912443-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x100003045ba0003
    2021-12-20 19:54:27,535 INFO  [master/challenge01:16000] zookeeper.ZooKeeper: Session: 0x3000030bfef0005 closed
    2021-12-20 19:54:27,535 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x3000030bfef0005
    2021-12-20 19:54:27,535 INFO  [master/challenge01:16000] regionserver.HRegionServer: Exiting; stopping=challenge01.e4motion.com,16000,1639997664082; zookeeper connection closed.
    2021-12-20 19:54:27,535 ERROR [main] master.HMasterCommandLine: Master exiting
    java.lang.RuntimeException: HMaster Aborted
            at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:261)
            at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:149)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
            at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
            at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3085)
    


    ------------------------------
    Cloudy tech
    ------------------------------