QuantumATK Forum

QuantumATK => General Questions and Answers => Topic started by: ziand on September 9, 2010, 11:21

Title: Damaged nc-file
Post by: ziand on September 9, 2010, 11:21

Hallo,

I did several subsequent calculations and wrote results to one big nc-file. My object_id's look like this:

Code

object_id= 'angle' + str(int(angle)).rjust(2, '0')

Angle is 0.0, 1.0, ... so I get object id's like 'angle-000', 'angle-001', ...

My nc-file is very big ~4.3GB. When I wanted to read a configuration I got the following error

Quote

  File "./zipdir/NL/IO/NLSaveUtilities.py", line 252, in nlread
  File "./zipdir/NL/IO/NetCDFFile.py", line 156, in __init__
  File "./zipdir/NL/IO/NetCDFFile.py", line 369, in _read
  File "./zipdir/NL/IO/NetCDFFile.py", line 440, in _read_var_array

If a have a wrong object_id, the output of nlread is an empty array []. But not such an error.
Did you see someting like this before? Could it be related to the file-size being too large?
Normally VNL is able to quickly print the object_id's of an nc-file. But it does not work here. Is there a way to find out the object-id's manually with atkpython?
Is this file unrepairably damaged or could there be a way to still get some data out of it?

Title: Re: Damaged nc-file
Post by: ziand on September 9, 2010, 11:23

I made a test within atkpython (interactive mode) and got basically the same error:

Quote

In [14]: x=nlread('Cu_5x5x5_CNT_6x0x2_Geom_185.nc',object_id='angle-000')
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
/home/ziand/atk/CNTDevice/src/<ipython console> in <module>()
/opt/QuantumWise/atk-10.8.0/atkpython/lib/python26.zip/NL/IO/NLSaveUtilities.pyc in nlread(filename, class_type, object_id, read_state)
/opt/QuantumWise/atk-10.8.0/atkpython/lib/python26.zip/NL/IO/NetCDFFile.pyc in __init__(self, filename, mode, mmap)
/opt/QuantumWise/atk-10.8.0/atkpython/lib/python26.zip/NL/IO/NetCDFFile.pyc in _read(self)
/opt/QuantumWise/atk-10.8.0/atkpython/lib/python26.zip/NL/IO/NetCDFFile.pyc in _read_var_array(self)
IOError: [Errno 22] Invalid argument

Title: Re: Damaged nc-file
Post by: Nordland on September 9, 2010, 12:37

The following code snippet gives the list of object id's in the file.

Code: python

from NL.IO.NLSaveUtilities import nlinspect
print nlinspect('your_file.nc')

However I think there is 2 possibilities that can cause this problem.

• The file is broken - caused by an interruption in saving or so - if you install the program ncdump and is able to see inside the file, it is likely that it is not broken.
• If the file is not broken, then file might be too large to read in. This can either be an issue in the code or it can be related to memory on the machine.

Title: Re: Damaged nc-file
Post by: Anders Blom on September 9, 2010, 13:15

"ncdump" is part of the NetCDF package from Unidata.

For Linux it can be obtained as pre-compiled binaries from http://www.unidata.ucar.edu/downloads/netcdf/index.jsp and for Windows at http://www.unidata.ucar.edu/software/netcdf/docs/faq.html (search for "Windows" on the page). Or, you can try to built it from source (http://www.unidata.ucar.edu/downloads/netcdf/index.jsp), if you have all the required libraries.

Title: Re: Damaged nc-file
Post by: ziand on September 9, 2010, 13:37

Thank you very much. The code for printing the object id's can be quite helpful with big files.

About my problem: Only one of my files is broken. The others are okay, I myself made a mistake there. So the problem is not very big for me. Anyway a installed ncdump and indeed was able to look inside the broken file. But before it gave me an error. After pressing "Ignore" two times it started dumping the file. The error message was on a small window from visual studio c++ runtime library:

Quote

Assertion failed!
Program ncdump
File: ..\..\..\libsrc\v1hpg.c
Line: 1157
Expression: ncp->begin_rec > 0

However, don't care about it as long as it does not happen more often. For now I have no problem here  ;).