Hi All,

In case we have a process that uploaded every day a file (append) to the Cognos Server and we made a mistake.
We uploaded a file double and want to remove a uploaded day source.

Can we delete some part records, or do we need to upload all (days) again ?

Kind Regards,

Chris

------------------------------
Chris Bos
------------------------------

#CognosAnalyticswithWatson

I'm glad to learn that you discovered the error in your process -- it would be sad if you had not discovered the error or only discovered it after a long time had passed.

One very important best practice for working with files as a data source would be to put the files into a source control environment. This will allow you to track what changes have taken place and to recover prior versions.

This, obviously, is helpful in cases like this but also will enable you prove to auditors etc. that you have that aspect of a data management system.

It would especially helpful if you have a case where a record ( a row of data ) is lost.

You might be able to fairly quickly observe that your data was wrong but there's always the case where the problem is not going to be obvious.

I don't immediately know how to have a system for files such that all changes can be knowable and problems similar to what you encountered could be prevented.

In general, I am concerned about people working with files as data sources without having a data management system in place.  A system, where there is a process which controls what data is uploaded in files, how it is uploaded, and  curates, documents and preserves the data independently, for record keeping and other purposes, such as support for audits should be implemented. It could be the case that data which is being used to make decisions could be invalid or mis-handled.  You might want to contemplate whether the data being captured in these files ought to be captured in a data base and subject to the controls usually associated with that.

As for the method of recovery, you have two options.

One would be to generate an excel file with your existing data (see the technique described in these threads) and then edit out the duplicated  records.  You would then use the replace option to replace the existing uploaded file.

The second option would be use your existing source files and replace the existing uploaded file with the first of the series of source files and append to it the other files in the series.

https://community.ibm.com/community/user/businessanalytics/communities/community-home/digestviewer/viewthread?GroupId=3061&MessageKey=28a30684-0782-4fcd-b682-37c9fb2e6fa4&CommunityKey=6b10df83-0b3c-4f92-8b1f-1fd80d0e7e58&tab=digestviewer

https://community.ibm.com/community/user/businessanalytics/communities/community-home/digestviewer/viewthread?MessageKey=03f3e641-f123-42d9-86f8-e1245fcb26aa&CommunityKey=6b10df83-0b3c-4f92-8b1f-1fd80d0e7e58&tab=digestviewer#bm03f3e641-f123-42d9-86f8-e1245fcb26aa

------------------------------
IAN HENDERSON
------------------------------

Original Message:
Sent: Fri October 02, 2020 01:17 PM
From: Chris Bos
Subject: Delete Records from uploaded files

Hi All,

In case we have a process that uploaded every day a file (append) to the Cognos Server and we made a mistake.
We uploaded a file double and want to remove a uploaded day source.

Can we delete some part records, or do we need to upload all (days) again ?

Kind Regards,

Chris

------------------------------
Chris Bos
------------------------------

#CognosAnalyticswithWatson

Hi Ian,

Thanks for your reply.

I understand your point of view and partly agree with you.
As a BI/DWH Architect I always want the data within a RDBMS......

But the world has changed and for me that's where 'Self-Service BI' is all about.
So I like the capability within Cognos Analytics end-users are able to upload files and create reports on top of that.
And yes, there will be issues with this, like double uploads.
And yes, I prefer putting this data in a database with the right process embedded instead of uploading.
But again: companies demands other capabilities like uploading without IT resources (with the risk making mistakes).
And when there are many mistakes and/or decisions made based on wrong data, there will be a solution: a DWH system.
DWH Systems are there with a reason and I have learned it need some time to get all people realize that.

But about my question: can your delete records for now is NO You can't.
I think it should be made possible (RFE?).

Kind Regards,

------------------------------
Chris Bos
------------------------------

Original Message:
Sent: Fri October 02, 2020 07:35 PM
From: IAN HENDERSON
Subject: Delete Records from uploaded files

I'm glad to learn that you discovered the error in your process -- it would be sad if you had not discovered the error or only discovered it after a long time had passed.

One very important best practice for working with files as a data source would be to put the files into a source control environment. This will allow you to track what changes have taken place and to recover prior versions.

This, obviously, is helpful in cases like this but also will enable you prove to auditors etc. that you have that aspect of a data management system.

It would especially helpful if you have a case where a record ( a row of data ) is lost.

You might be able to fairly quickly observe that your data was wrong but there's always the case where the problem is not going to be obvious.

I don't immediately know how to have a system for files such that all changes can be knowable and problems similar to what you encountered could be prevented.

In general, I am concerned about people working with files as data sources without having a data management system in place.  A system, where there is a process which controls what data is uploaded in files, how it is uploaded, and  curates, documents and preserves the data independently, for record keeping and other purposes, such as support for audits should be implemented. It could be the case that data which is being used to make decisions could be invalid or mis-handled.  You might want to contemplate whether the data being captured in these files ought to be captured in a data base and subject to the controls usually associated with that.

As for the method of recovery, you have two options.

One would be to generate an excel file with your existing data (see the technique described in these threads) and then edit out the duplicated  records.  You would then use the replace option to replace the existing uploaded file.

The second option would be use your existing source files and replace the existing uploaded file with the first of the series of source files and append to it the other files in the series.

https://community.ibm.com/community/user/businessanalytics/communities/community-home/digestviewer/viewthread?GroupId=3061&MessageKey=28a30684-0782-4fcd-b682-37c9fb2e6fa4&CommunityKey=6b10df83-0b3c-4f92-8b1f-1fd80d0e7e58&tab=digestviewer

https://community.ibm.com/community/user/businessanalytics/communities/community-home/digestviewer/viewthread?MessageKey=03f3e641-f123-42d9-86f8-e1245fcb26aa&CommunityKey=6b10df83-0b3c-4f92-8b1f-1fd80d0e7e58&tab=digestviewer#bm03f3e641-f123-42d9-86f8-e1245fcb26aa

------------------------------
IAN HENDERSON

Original Message:
Sent: Fri October 02, 2020 01:17 PM
From: Chris Bos
Subject: Delete Records from uploaded files

Hi All,

In case we have a process that uploaded every day a file (append) to the Cognos Server and we made a mistake.
We uploaded a file double and want to remove a uploaded day source.

Can we delete some part records, or do we need to upload all (days) again ?

Kind Regards,

Chris

------------------------------
Chris Bos
------------------------------

#CognosAnalyticswithWatson